Repository navigation
Conversation
Overall package sizeSelf size: 3.01 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | pprof-format | 2.4.0 | 912.36 kB | 912.36 kB | | source-map | 0.8.0 | 185.66 kB | 185.66 kB | | node-gyp-build | 4.8.4 | 13.86 kB | 13.86 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
szegedi
force-pushed
the
szegedi/otel-builder-extensions
branch
from
November 12, 2026 12:13
b00c035 to
6f9e989
Compare
szegedi
force-pushed
the
szegedi/otel-serializer
branch
from
November 12, 2026 12:13
1cca0e8 to
6d9b255
Compare
|
🔗 Commit SHA: 89b577d | Docs | View more details | Give us feedback! |
OtelProfilesBuilder collects any number of normalized profiles, such as the time and heap profiles of one collection cycle, into one ExportProfilesServiceRequest of pprof-format/otel. All of them share the request's dictionary, as the OTel format intends. Each profile is added through a ProfileVisitor from visitor(), so whatever drives the visit, the normalizer or a profiler emitting samples itself, needs no knowledge of OTel. serializeTimeProfileOtel() and serializeHeapProfileOtel() are shorthands for a request with a single profile. - An OTel profile has a single sample type, so each added profile becomes one OTel profile per sample type (sample count, wall and CPU time; in-use and allocated objects and space). When every sampling event is a single timestamped observation, as with samplePerContext, the sample count is folded into the profile of the period type (wall time) instead, provided no other sample adds an observation to that profile. It then gets an observation for every sampling event, even one whose value is zero, so the count is its number of observations. Idle samples, which deliberately have zero wall time (see #199), thereby stay in the wall profile as timestamped evidence that sampling happened, while their CPU time stays in the CPU profile. No count profile then has to repeat every sample just to carry 1s. When some sampling events are summed, the count keeps its profile, so no count is lost. - The profiles' time range starts at the start of collection, so sample timestamps fall within [time_unix_nano, time_unix_nano + duration_nano), as the format asks. All profiles of a request get the union of their ranges, so one collection cycle's profiles cover the same period, including a heap profile, which is a snapshot at one instant. - Frames become locations, with columns, and the normalizer's stacks become the stack table. Functions are identified by what they encode as, display name and resolved file, so one script mapping to several original files keeps them apart. Functions, locations, stacks, attributes and links are deduplicated across all profiles. Every dictionary table starts with its zero value. - Labels become interned attributes: strings, integers, or doubles for fractional numbers, which pprof labels can't hold. An optional extractLink callback can turn some labels into a link to a span; the builder validates the ID lengths, copies the IDs, and treats a link whose trace ID or span ID is all zeros, which is invalid, as no link. - A sample whose stack has no frames, such as a timeline event, refers to the dictionary's zero-value stack. - Samples with the same stack, attributes and link are merged. Timestamped samples accumulate a value and a timestamp per observation; others are summed. Other zero values, such as zero CPU time, are left out of their sample type's profile. - Frames whose source map is declared but missing locally, the candidates for server-side unminification, get a datadog.frame.missing_source_map location attribute. The profiles with such frames get a datadog.profile.has_missing_source_maps attribute, which the builder learns from the visitor's end() callback. pprof carries this as a profile comment, but OTel's pprof.profile.comment is reserved for human-readable text. No semantic convention exists for this, so the keys are provisional until the Datadog backend confirms them. visitor() takes options for the OTel profiles that the visit adds: attributes, and an original payload with its format. Attribute values can be structured, e.g. key-value lists, and are interned by a canonical form of their value, so repeating per-collection data such as the kind of snapshot or endpoint counts on each profile costs an index each. The original payload can carry a document the backend reads while a representation in attributes is agreed; the proto requires its two fields to be set together, so they're set as one. Visitors run synchronously, and build() can be called after any number of completed visits; a visit that throws leaves partial data, so the builder should then be discarded. The builder, its options and the shorthands are exported from the package. pprof-format/otel first shipped in pprof-format 2.4.0, so that is now the minimum version.
These pass the normalized profile to a caller-supplied ProfileVisitor, instead of serializing it as pprof. The caller chooses the output, e.g. a visitor of an OtelProfilesBuilder that collects all profiles of a collection cycle into one request. They are the format-neutral counterparts of time.stop() and heap.profile(). Both use the eager stop paths of time.stop() and heap.profile(), not the lazy, callback-based ones, which are opt-in because of a race in V8 when deleting a CPU profile (#329). Both return the VisitResult.
PprofProfileBuilder is the pprof counterpart of OtelProfilesBuilder: its visitor() collects one normalized profile, and build() returns it as a pprof Profile. A profiler that emits samples itself, such as a timeline of events, can then produce pprof through the same visitor it uses for OTel, instead of building pprof objects by hand. serializeTimeProfile() and serializeHeapProfile() now use it, and their output is unchanged. The builder and its options are exported from the package. The missing source map comment comes from the visitor's end() call, which reports the same result the normalizer returns. pprof samples have no timestamps, so sample timestamps aren't encoded; a profiler that needs them carries them as labels.
teeVisitor() passes a single visit on to several visitors, so that one collection can produce a profile in more than one format, e.g. pprof and OTel while the two are compared. Each visitor returns its own frame and stack IDs, so the tee hands out IDs of its own and translates them for each visitor. It is exported from the package.
szegedi
force-pushed
the
szegedi/otel-serializer
branch
from
November 12, 2026 12:40
6d9b255 to
a136cc8
Compare
szegedi
force-pushed
the
szegedi/otel-builder-extensions
branch
from
November 12, 2026 12:40
6f9e989 to
a6a083c
Compare
pprof samples have no timestamps. A PprofProfileBuilder created with a timestampLabel now adds a numeric label of that name with each sample's timestamp, e.g. end_timestamp_ns, as dd-trace-js does for its timeline. A visitor can then deliver timestamps the same way for pprof and OTel, instead of adding the label itself for pprof. The label is unique to its sample, so it isn't shared like low-cardinality labels. Without the option, timestamps aren't encoded, as before.
szegedi
marked this pull request as ready for review
November 13, 2026 17:12
szegedi
requested review from
IlyasShabi,
nsavoire and
r1viollet
as code owners
November 13, 2026 17:12
szegedi
force-pushed
the
szegedi/otel-serializer
branch
from
November 13, 2026 17:29
a136cc8 to
3d5ece6
Compare
Collaborator
Author
|
Merged into #437, whose commits now include the pprof profile builder (with |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?:
Adds two visitor utilities on top of #437:
PprofProfileBuilder: the pprof counterpart ofOtelProfilesBuilder. WithtimestampLabel, it encodes sample timestamps as a label of that name (e.g.end_timestamp_ns), since pprof samples have none.teeVisitor(...visitors): delivers one visit to several visitors, translating frame and stack IDs for each.Motivation:
Both are needed for the upcoming OTel support in dd-trace-js:
PprofProfileBuilderit can emit through a visitor, and the same code produces pprof or, throughOtelProfilesBuilder, OTel.teeVisitorlets one collection produce both formats, so pprof and OTel output can be compared while OTel support is rolled out.Additional Notes:
end()call, which reports the same result the normalizer returns.