Skip to content

fix: map the pcc hierarchy from the top ancestor instead of dropping it - #5345

Open
ulemons wants to merge 1 commit into
mainfrom
fix/issue-984-pcc-hierarchy-depth
Open

ulemons wants to merge 1 commit into
mainfrom
fix/issue-984-pcc-hierarchy-depth

Conversation

@ulemons

@ulemons ulemons commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Summary

The PCC sync compared every project against the wrong CDP group. The parser assumed every PCC tree starts at The Linux Foundation and dropped the top level. In PROJECT_SPINE the foundations are at the top, and only Community Projects sit under The Linux Foundation, so most projects were reported as HIERARCHY_MISMATCH while CDP had them in the right place.

  • rowParser.ts: one rule for any depth. The group is the top ancestor, the project is the level below it (the top ancestor itself for a project with no parent), and the sub-project is the leaf. Community Projects need no special case: their top ancestor is "The Linux Foundation", which is the name of the CDP tlf group.
  • rowParser.ts: a leaf with no PROJECT_SPINE rows came through the LEFT JOIN with a null level, parsed as 0, and was reported as "unexpected root". It is now reported as having no valid hierarchy rows.
  • types.ts, index.ts: effectiveDepth and mappingRule are removed. Nothing read them.
  • pccProjectConsumer.ts: comments only, they described the old depth rules.
  • rowParser.test.ts: first tests of this worker. Trees of 1 to 5 levels, a Community Project, unordered rows, a level serialized as a Parquet buffer, leaf metadata, and the three error cases.

Effect

Both parsers were run locally on the Snowflake export and the CDP segments of 2026-11-13 (1,921 leaf projects). Nothing was written. The old parser reproduces last night's production run exactly.

Result of a run Before After
Synced, no mismatch 96 942
Synced, with HIERARCHY_MISMATCH 983 155
Missing in CDP 712 803
SCHEMA_MISMATCH 130 21
  • 876 false mismatches are gone.
  • The 109 projects with no parent in PCC are no longer errors. 18 of them exist in CDP and start getting their metadata synced; 91 are missing in CDP.
  • Three projects change name on the first run: CTi → Core Toolchain Infrastructure (CTI), Murmur Project → OpenSTX Foundation, and C2PA gets its acronym.
  • 41 projects that showed no mismatch now show one. PCC has them under The Linux Foundation and CDP has them under their own group (O-RAN Software Community, Project Jupyter, Joint Development Foundation and others). The old parser hid this by dropping the top level. The sync only records the mismatch; it never moves a project.

Not in this PR

  • Membership children of The Linux Foundation that the spine leaves under it (31 projects, one of them in CDP today): linuxfoundation/lfx-cdp-insights-ops#1142.
  • Clearing the error rows that a run no longer reports: linuxfoundation/lfx-cdp-insights-ops#985.

Verification

  • pnpm exec vitest run --project server services/apps/pcc_sync_worker/src/parser/rowParser.test.ts: 11 passed
  • pnpm lint --deny-warnings --format=agent, pnpm tsc-check: clean
  • After deploy: run the worker with PCC_DRY_RUN=true and compare the PCC job completed log with the table above.

Refs linuxfoundation/lfx-cdp-insights-ops#984

🤖 Generated with Claude Code

The parser assumed every PCC tree starts at The Linux Foundation and
dropped the top level, so projects under a foundation were compared
against the wrong group. The group is now the top ancestor of the spine,
the project is the level below it, and the leaf is the subproject, for
any depth. Leaves with no spine rows are reported as such instead of as
unexpected roots.

Refs linuxfoundation/lfx-cdp-insights-ops#984

Signed-off-by: Umberto Sgueglia <[email protected]>
@ulemons ulemons self-assigned this Nov 13, 2026
Copilot AI balanced review requested due to automatic review settings November 13, 2026 16:23
@cursor

cursor Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes core PCC→CDP hierarchy interpretation used for sync and HIERARCHY_MISMATCH reporting across the full project export; writes still avoid re-parenting, but mismatch and schema-error classification will shift materially at deploy.

Overview
Replaces the PCC row parser’s effective-depth / four-rule mapping (which stripped the top ancestor and assumed a TLF root) with a single rule: CDP group = top HIERARCHY_LEVEL, project = level just below the top (or the leaf when standalone), subproject = leaf. Deep trees no longer fail as “unsupported depth”; intermediate levels are collapsed into that trio.

Malformed spine rows are handled more accurately: invalid levels below 1 are dropped (e.g. null PROJECT_SPINE from the LEFT JOIN), and missing ancestor names produce explicit SCHEMA_MISMATCH reasons. effectiveDepth, mappingRule, and MappingRule are removed from parser types/exports. pccProjectConsumer only updates comments to match the new model.

Adds rowParser.test.ts covering 1–5 level trees, Community Projects under TLF, row order, Parquet buffer levels, metadata trimming, and error paths.

Reviewed by Cursor Bugbot for commit 86ec6a3. Bugbot is set up for automated code reviews on this repo. Configure here.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the PCC-to-CDP hierarchy-mapping update.

No open issues 🎉

What changed in this PR

Corrects PCC-to-CDP hierarchy mapping so the sync uses the top ancestor as the group instead of always discarding it. This addresses false hierarchy mismatches while preserving the sync’s behavior of reporting mismatches without moving projects.

Changes:

  • Maps the top ancestor, its immediate child, and the leaf to CDP’s group, project, and subproject levels, including standalone projects.
  • Rejects unusable hierarchy levels and removes unused effectiveDepth and mappingRule fields.
  • Adds parser tests for hierarchy mapping, row ordering, buffer-encoded levels, metadata, and error cases.
File Description
services/​apps/​pcc_sync_worker/​src/​parser/​types.ts Removes obsolete depth-mapping types and fields; updates error-field comments.
services/​apps/​pcc_sync_worker/​src/​parser/​rowParser.ts Implements top-ancestor mapping and tightens hierarchy-level validation.
services/​apps/​pcc_sync_worker/​src/​parser/​rowParser.test.ts Adds 11 tests covering mapping, input handling, metadata, and errors.
services/​apps/​pcc_sync_worker/​src/​parser/​index.ts Removes the obsolete MappingRule export.
services/​apps/​pcc_sync_worker/​src/​consumer/​pccProjectConsumer.ts Updates comments to reflect the new hierarchy rules; no functional changes.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants