Skip to content

feat: add an OpenTelemetry profiles serializer - #437

Draft
szegedi wants to merge 4 commits into
mainfrom
szegedi/otel-serializer
Draft

szegedi wants to merge 4 commits into
mainfrom
szegedi/otel-serializer

Conversation

@szegedi

@szegedi szegedi commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?:

Adds an OpenTelemetry profiles serializer, built as a ProfileVisitor on the normalizer from #430, and encoding with pprof-format/otel (pprof-format 2.4.0).

Multiple profiles (both time and heap) can be collected into a single ExportProfilesServiceRequest sharing the request's dictionary.
Each profile is added through builder.visitor(), so whatever drives the visit needs no knowledge of OTel. serializeTimeProfileOtel() and serializeHeapProfileOtel() are shorthands for a single-profile request.
builder.visitor() takes attributes and an original payload for the profiles of that visit, for per-collection metadata (e.g. event.json in dd-trace-js).

time.stopAndVisit() and heap.visitProfile() pass the normalized profile to a caller-supplied visitor instead of serializing it as pprof. They're the format-neutral counterparts of time.stop() and heap.profile()

PprofProfileBuilder is the pprof counterpart of OtelProfilesBuilder; serializeTimeProfile() and serializeHeapProfile() now use it, with unchanged output. With timestampLabel it encodes sample timestamps as a label (e.g. end_timestamp_ns), since pprof samples have none.
teeVisitor(...visitors) delivers one visit to several visitors, so one collection can produce both pprof and OTel while they're compared. (Both were #441.)

Motivation:

Step 2 of emitting the OpenTelemetry profiles signal from the Node.js profiler: a second encoder that reuses the profile semantics from #430 instead of duplicating them.

Additional notes:

  • One OTel profile is emitted per sample type (sample count, wall and CPU time; in-use and allocated objects and space).
  • With samplePerContext, the sample count is folded into the wall-time profile, which gets an observation per sampling event. Idle samples, which deliberately have zero wall time (see Preserve idle samples with zero wall time and no CPU time rollup #199), stay as timestamped evidence that sampling happened, and we don't have to have a separate count profile that has to repeat every sample just to carry 1s.
  • Labels become interned attributes: strings, integers, or doubles for fractional numbers. An optional extractLink callback can turn labels into a span link; all-zero trace or span IDs are treated as no link.
  • Samples with the same stack, attributes and link are merged: timestamped ones accumulate a value and timestamp per observation, others are summed.
  • Samples without frames (e.g. timeline events) get the dictionary's zero-value stack.
  • All profiles in a request get the same time range (the union of their own), so a collection cycle's profiles cover the same period, including heap snapshots.
  • The first commit moves the existing pprof serialization onto PprofProfileBuilder; its output is unchanged (the existing fixture tests and a byte comparison with serializeTimeProfile()).
  • Frames whose source map is declared but missing locally get a datadog.frame.missing_source_map location attribute, and their profiles a datadog.profile.has_missing_source_maps attribute. (pprof carries this as a profile comment, for lack of better solution.) The backend will need to be adjusted to recognize these.
AI generated PR description Adds an **OpenTelemetry profiles serializer**, built as a `ProfileVisitor` on the format-neutral normalizer from #430, and encoding with `pprof-format/otel` (pprof-format 2.4.0).
  • OtelProfilesBuilder (ts/src/otel-serializer.ts) collects any number of normalized profiles, e.g. the time and heap profiles of one collection cycle, into one ExportProfilesServiceRequest. All of them share the request's dictionary, as the OTel format intends. Each profile is added through builder.visitor(), so whatever drives the visit needs no knowledge of OTel. serializeTimeProfileOtel() and serializeHeapProfileOtel() are shorthands for a single-profile request.
  • time.stopAndVisit() and heap.visitProfile() pass the normalized profile to a caller-supplied visitor instead of serializing it as pprof. They're the format-neutral counterparts of time.stop() and heap.profile(), and use the same eager stop paths. The lazy, callback-based paths stay opt-in because of the V8 CpuProfile deletion race (Revert lazy profile-tree changes to mitigate CpuProfile::Delete crash #329).
  • PprofProfileBuilder (ts/src/profile-serializer.ts) builds pprof through the same visitor interface; serializeTimeProfile()/serializeHeapProfile() use it. Its timestampLabel option encodes sample timestamps as a numeric label. teeVisitor() (ts/src/profile-normalizer.ts) feeds one visit to several visitors, translating frame and stack IDs for each.
  • builder.visitor({attributes, originalPayload}) sets attributes and an original payload on every OTel profile the visit adds. Attribute values can be structured (kvlistValue, arrayValue) and are interned by a canonical form of their value, so per-collection metadata such as the snapshot kind or endpoint counts costs an index per profile. The original payload carries a document the backend reads while an attribute representation is agreed (for now, event.json); its format and bytes are set together, as the proto requires.

Mapping highlights:

  • One OTel profile per sample type (sample count, wall and CPU time; in-use and allocated objects and space).
  • With samplePerContext, the sample count is folded into the wall-time profile, which gets an observation per sampling event. Idle samples, which deliberately have zero wall time (see Preserve idle samples with zero wall time and no CPU time rollup #199), thereby stay as timestamped evidence that sampling happened, and no count profile has to repeat every sample just to carry 1s.
  • Sample timestamps fall within [time_unix_nano, time_unix_nano + duration_nano), since the time range starts at the start of collection.
  • Functions, locations (with columns), stacks, attributes and links are deduplicated across all profiles. Every dictionary table starts with its zero value.
  • Labels become interned attributes: strings, integers, or doubles for fractional numbers. An optional extractLink callback can turn labels into a span link; all-zero trace or span IDs are treated as no link.
  • Samples with the same stack, attributes and link are merged: timestamped ones accumulate a value and timestamp per observation, others are summed.
  • A sample whose stack has no frames, such as a timeline event, refers to the dictionary's zero-value stack.

Motivation:

Step 2 of emitting the OpenTelemetry profiles signal from the Node.js profiler: a second encoder that reuses the profile semantics from #430 instead of duplicating them.

Additional Notes:

  • Based on main, now that refactor: extract a format-neutral profile normalizer from the pprof serializer #430 is merged.
  • OtelProfilesBuilder, its options, the shorthands and the missing-source-map attribute keys are exported from the package, for dd-trace-js.
  • Frames whose source map is declared but missing locally get a datadog.frame.missing_source_map location attribute, and their profiles a datadog.profile.has_missing_source_maps attribute. pprof carries this as a profile comment, but OTel's pprof.profile.comment is reserved for human-readable text. No semantic convention exists, so these keys are provisional until the Datadog backend confirms them.
  • Visitors run synchronously. A visit that throws leaves partial data, so the builder should then be discarded.

How to test the change?:

Unit tests in ts/test/test-otel-serializer.ts cover the mapping, deduplication across profiles, sample merging, links, visit attributes and original payloads, samples without frames, the missing-source-map attributes and the package exports, and decode the encoded request to check it round-trips. ts/test/test-profile-serializer.ts covers PprofProfileBuilder, including timestamp labels and byte-identical output to serializeTimeProfile(); a teeVisitor test checks that one tee'd visit gives the same pprof and OTel bytes as separate visits. ts/test/test-time-profiler.ts and ts/test/test-heap-profiler.ts cover stopAndVisit() and visitProfile(), including state after a throwing visitor. npm test.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Overall package size

Self size: 3.01 MB
Deduped: 4.12 MB
No deduping: 4.12 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | pprof-format | 2.4.0 | 912.36 kB | 912.36 kB | | source-map | 0.8.0 | 185.66 kB | 185.66 kB | | node-gyp-build | 4.8.4 | 13.86 kB | 13.86 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@szegedi
szegedi force-pushed the szegedi/otel-serializer branch from c6ec787 to 59f320e Compare November 10, 2026 11:36
Base automatically changed from szegedi/profile-normalizer to main November 12, 2026 10:06
@szegedi
szegedi force-pushed the szegedi/otel-serializer branch from 59f320e to 1cca0e8 Compare November 12, 2026 11:11
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Pipelines

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 4 Pipeline jobs failed

Build | build / alpine-test-26 — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

Build | build / linux-x64-test-26 — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

Build | build / win32-test-26

View more details · View in GitHub Actions

View all 4 failed jobs.

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3d5ece6 | Docs | View more details | Give us feedback!

@szegedi szegedi added the semver-minor Usually minor non-breaking improvements label Nov 12, 2026
@szegedi
szegedi force-pushed the szegedi/otel-serializer branch 2 times, most recently from 6d9b255 to a136cc8 Compare November 12, 2026 12:40
PprofProfileBuilder builds a pprof profile through a ProfileVisitor: its
visitor() collects one normalized profile, and build() returns it as a
pprof Profile. A profiler that emits samples itself, such as a timeline
of events, can then produce pprof through a visitor, as it would any
other format, instead of building pprof objects by hand.

serializeTimeProfile() and serializeHeapProfile() now use it, and their
output is unchanged. The builder and its options are exported from the
package. The missing source map comment comes from the visitor's end()
call, which reports the same result the normalizer returns.

pprof samples have no timestamps. With a timestampLabel, the builder
adds a numeric label of that name with each sample's timestamp, e.g.
end_timestamp_ns, as dd-trace-js does for its timeline, so a visitor can
deliver timestamps the same way whatever the format. The label is unique
to its sample, so it isn't shared like low-cardinality labels. Without
the option, timestamps aren't encoded.
OtelProfilesBuilder collects any number of normalized profiles, such as
the time and heap profiles of one collection cycle, into one
ExportProfilesServiceRequest of pprof-format/otel. All of them share the
request's dictionary, as the OTel format intends. Each profile is added
through a ProfileVisitor from visitor(), so whatever drives the visit,
the normalizer or a profiler emitting samples itself, needs no knowledge
of OTel. serializeTimeProfileOtel() and serializeHeapProfileOtel() are
shorthands for a request with a single profile.

- An OTel profile has a single sample type, so each added profile
  becomes one OTel profile per sample type (sample count, wall and CPU
  time; in-use and allocated objects and space). When every sampling
  event is a single timestamped observation, as with samplePerContext,
  the sample count is folded into the profile of the period type (wall
  time) instead, provided no other sample adds an observation to that
  profile. It then gets an observation for every sampling event, even
  one whose value is zero, so the count is its number of observations.
  Idle samples, which deliberately have zero wall time (see #199),
  thereby stay in the wall profile as timestamped evidence that sampling
  happened, while their CPU time stays in the CPU profile. No count
  profile then has to repeat every sample just to carry 1s. When some
  sampling events are summed, the count keeps its profile, so no count
  is lost.
- The profiles' time range starts at the start of collection, so sample
  timestamps fall within [time_unix_nano, time_unix_nano +
  duration_nano), as the format asks. All profiles of a request get the
  union of their ranges, so one collection cycle's profiles cover the
  same period, including a heap profile, which is a snapshot at one
  instant.
- Frames become locations, with columns, and the normalizer's stacks
  become the stack table. Functions are identified by what they encode
  as, display name and resolved file, so one script mapping to several
  original files keeps them apart. Functions, locations, stacks,
  attributes and links are deduplicated across all profiles. Every
  dictionary table starts with its zero value.
- Labels become interned attributes: strings, integers, or doubles for
  fractional numbers, which pprof labels can't hold. An optional
  extractLink callback can turn some labels into a link to a span; the
  builder validates the ID lengths, copies the IDs, and treats a link
  whose trace ID or span ID is all zeros, which is invalid, as no link.
- A sample whose stack has no frames, such as a timeline event, refers
  to the dictionary's zero-value stack.
- Samples with the same stack, attributes and link are merged.
  Timestamped samples accumulate a value and a timestamp per
  observation; others are summed. Other zero values, such as zero CPU
  time, are left out of their sample type's profile.
- Frames whose source map is declared but missing locally, the
  candidates for server-side unminification, get a
  datadog.frame.missing_source_map location attribute. The profiles
  with such frames get a datadog.profile.has_missing_source_maps
  attribute, which the builder learns from the visitor's end()
  callback. pprof carries this as a profile comment, but OTel's
  pprof.profile.comment is reserved for human-readable text. No
  semantic convention exists for this, so the keys are provisional
  until the Datadog backend confirms them.

visitor() takes options for the OTel profiles that the visit adds:
attributes, and an original payload with its format. Attribute values
can be structured, e.g. key-value lists, and are interned by a canonical
form of their value, so repeating per-collection data such as the kind
of snapshot or endpoint counts on each profile costs an index each. The
original payload can carry a document the backend reads while a
representation in attributes is agreed; the proto requires its two
fields to be set together, so they're set as one.

Visitors run synchronously, and build() can be called after any number
of completed visits; a visit that throws leaves partial data, so the
builder should then be discarded. The builder, its options and the
shorthands are exported from the package.

pprof-format/otel first shipped in pprof-format 2.4.0, so that is now
the minimum version.
These pass the normalized profile to a caller-supplied ProfileVisitor,
instead of serializing it as pprof. The caller chooses the output, e.g.
a visitor of an OtelProfilesBuilder that collects all profiles of a
collection cycle into one request. They are the format-neutral
counterparts of time.stop() and heap.profile().

Both use the eager stop paths of time.stop() and heap.profile(), not
the lazy, callback-based ones, which are opt-in because of a race in V8
when deleting a CPU profile (#329). Both return the VisitResult.
teeVisitor() passes a single visit on to several visitors, so that one
collection can produce a profile in more than one format, e.g. pprof and
OTel while the two are compared. Each visitor returns its own frame and
stack IDs, so the tee hands out IDs of its own and translates them for
each visitor. It is exported from the package.
@szegedi
szegedi force-pushed the szegedi/otel-serializer branch from a136cc8 to 3d5ece6 Compare November 13, 2026 17:29

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

semver-minor Usually minor non-breaking improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant