Skip to content

Read Cargo members in discovery; match liveness uuids by origin - #2012

Open
Mikola Lysenko (mikolalysenko) wants to merge 5 commits into
mainfrom
fix/cargo-members-liveness-origin
Open

Mikola Lysenko (mikolalysenko) wants to merge 5 commits into
mainfrom
fix/cargo-members-liveness-origin

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes two cases where VEX discovery or ledger liveness vouched for wiring the build doesn't use.

C2: Cargo discovery ignored workspace members

Hosted mode pins a crate in the root Cargo.toml and in every workspace-member manifest that declares it (rewrite_cargo_with). Vendored mode's live-redirect refusal reads the member manifests too. Discovery read only the root manifest. When the root didn't declare a crate, it treated the lock as the only word and attested the ref. So if a member's declaration was reverted to crates.io, the ref stayed attested even though cargo re-resolves that dependency from crates.io.

  • Discovery now finds the members the same way hosted does: cargo_workspace::member_manifests_in, filtered by the rewriter's own is_cargo_member_manifest_key (now pub(crate)), so nothing under .socket/ or target/ counts. It then runs its existing declaration check over the root and every member.
  • Each declaration keeps its file. Diagnostics, the conflict-pair Cargo.toml ref and lockless UnlockedPins now name the member manifests that hold the pin.
  • [patch] is still read from the root only, because cargo ignores a member's.

P6: ledger liveness matched the uuid as a substring

Step 1 of redirect_record_live treated a record as live if any inventory entry's resolved URL merely contained the uuid. Step 1 now runs the URL through patch::redirect::hosted_patch_uuid with Socket's host plus the run's patch-server origins, and requires the uuid it returns to equal the record's.

The origins come from the new field Discovery::patch_server_origins, which discover_with_ctx fills from its ctx. Liveness therefore uses exactly the origins discovery used. A Socket-host URL or a configured-origin URL for the record's uuid is Live. For the other cases, see review finding 2 below.

Review findings (/code-review high)

All findings were confirmed. For findings 1, 3, 5 and 7, a reduced failing test was run against the previous PR head 1a043ec, and all four failed. Finding 2 was already shown by the earlier test, where a configured-origin URL with no origin passed was judged dead. Fixes are in 650143e and b2e171b.

  1. Lockless pin suppressed by a member's other-version declaration: fixed.
    • UnlockedPin gains excluded_reqs, the requirements of the crate's unpinned declarations. The rewriter leaves a declaration alone only when its requirement excludes the patched version. So a ledger version is routed when it satisfies every pinned requirement and no excluded one.
    • A member's serde = "0.9" next to the root's pinned serde = "1" keeps the pin. A member serde = "1.0" does not, and neither does a member declaration with no requirement at all (git or path).
    • Test: lockless_pin_survives_a_member_declaring_another_version.
  2. Records from a mirrored or configured patch server became dead when a later run lacked the origin: fixed with a three-state verdict.
    • New RecordLiveness { Live, Undecided, Dead } and redirect_record_verdict.
    • A URL with the hosted shape for the record's uuid on an origin this run wasn't given is Undecided. "Hosted shape" means hosted_patch_uuid's grammar with the URL's own origin taken as configured.
    • Undecided is never attested: redirect_record_live and LedgerLiveness::redirect_record stay attestation-grade, so vex is unchanged.
    • Undecided is never judged dead either. scan's takeover classifier and hosted-wiring-retained advisory now use LedgerLiveness::redirect_record_retained.
    • A URL that merely contains the uuid (in a file name or a query) is still Dead, as is a Socket URL for another patch.
    • Why not record the origin in the ledger: the record is the shared PatchRecord (manifest schema). The ledger never deletes records on this verdict; the only consumers are attestation and the two scan advisories. A three-state verdict fixes those without a schema migration, and it covers ledgers written before such a field existed.
    • Test: liveness_matches_the_resolved_url_by_origin_not_substring. It checks seven URL/origin cases across the verdict, attest and retained APIs.
  3. N+1 lockless pins and diagnostics: fixed.
    • There is now one UnlockedPin per (crate, patch). It carries the union of the pinning declarations' requirements; its file is the first pinning manifest, in root-first order.
    • There is one DIAG_REF_UNATTRIBUTABLE per (crate, patch), listing each pinning manifest once. Malformed socket-patch-* names are diagnosed once per (file, name).
    • Test: one_lockless_pin_and_diagnostic_per_crate_and_patch.
  4. SnapshotTree bypassed the view: fixed.
    • The walk now goes through the ProjectView. Manifests are read with read_text, globs list with list_dir, and directory probes use the view's own mediated is_dir and is_symlink.
    • Overlays, the content cache and recording therefore come for free. DiskSnapshot::touch and touch_listing are private again, and SnapshotTree is gone.
    • The glob expansion is an iterative twin of expand_from that shares the segment, child-filter and match helpers. view_walk_matches_the_disk_walk pins it to the disk walk over globs, **, excludes, path dependencies, [patch] copies, dot and target dirs, and a symlinked member.
    • Tests: snapshot_walk_sees_overlays and view_walk_matches_the_disk_walk.
  5. Perf: fixed.
    • Members are walked only when the lock has a Socket-sourced package, or when there is no lock.
    • With no lock, the members are parsed only if the root or a member text carries socket-patch-.
    • An unparseable lock skips the walk, since it produces no ref either way.
    • Test: members_are_read_only_when_socket_wiring_could_need_them.
  6. Sync walk inside async discovery: fixed.
    • member_manifests_in is async. Each level's manifests are read concurrently, bounded at 16 with ordered_concurrent, so results stay deterministic.
    • The walk returns the texts it read, and discovery adopts them through DiscoverCtx::adopt_read (read log plus rule-11 recognition) instead of reading them again.
    • Directory probes remain the view's synchronous stat calls, as the view exposes them everywhere else.
    • The in-memory engine keeps the sync walk (member_manifests_in_memory), and disk-only callers keep member_manifests.
  7. Racy read set: fixed. Reads now go through the snapshot cache, so freshly written manifests hold. That includes a .socket/ copy followed through [patch], which the walk still visits as the rewriter does. Test: snapshot_walk_records_what_it_reads, which uses fresh files with no mtime ageing.
  8. Root DocumentMut deep clone: fixed. The root is moved into the manifest list and read back as manifests.first().

Tests

Checks run:

  • cargo test -p socket-patch-core --lib -- vex::discover:: vendor::cargo patch::redirect:: utils::cargo_workspace hosted::: 1733 passed.
  • The golden test committed_fixture_corpus_matches_golden passes with no golden changes. excluded_reqs is rendered only when non-empty.
  • CLI: commands::vex_sources and commands::scan (242), plus mode_migration_npm (15), in_process_vendor_pypi_takeover (7) and in_process_vendor_pnpm_takeover (32), all pass.
  • cargo clippy --locked --workspace --all-features -- -D warnings and cargo fmt --all -- --check are clean.

Not done (follow-ups)

  • Mixed member states. If the root or a member pins the patch and another member declares the crate unpinned at a version the pin covers, the ref is still attested (main behaves the same). The right fix is to keep the ref and mark it unattested with an UnattestedKind. That is tracked by the stacked Stop attesting a Cargo pin a workspace member resolves from crates.io #2019.
  • Liveness step 3. hosted_wiring_in_files, the fallback over ledger-recorded files, still matches the uuid as raw text, with role checks.

Note

Medium Risk
Touches Cargo discovery, attestable refs, and redirect-ledger liveness semantics; behavior changes are nuanced but heavily test-backed.

Overview
Cargo discovery now enumerates every workspace-member Cargo.toml the hosted rewriter would touch (async ProjectView walk, concurrent reads, same filters as disk: skip .socket/ and target/), merges those declarations with the root, attributes diagnostics and conflict refs to the real manifest path, collapses lockless pins to one per crate+patch with combined version_reqs, and tracks excluded_reqs from unpinned declarations so mixed root/member version lines keep or invalidate a pin correctly.

Redirect-ledger liveness no longer treats a resolved URL that merely contains the uuid as live: it normalizes URLs like hosted patch parsing, compares against Socket's host plus the run's patch_server_origins, introduces RecordLiveness { Live, Undecided, Dead }, and splits APIs so attestation stays Live-only while scan advisories use redirect_record_retained (Undecided wiring is kept, not attested). Discovery now carries those origins for liveness.

Infra: member_manifests_in is async with shared glob logic; member_manifest_texts_in returns manifest text so discovery adopt_reads without rereading; in-memory keeps a sync path; normalized_hosted_url is factored for uuid extraction; golden output optionally includes excluded_reqs.

Extensive new tests cover member-only reverts, lockless member pins, perf skips, snapshot parity, and origin-based liveness cases.

Reviewed by Cursor Bugbot for commit 94af3e4. Configure here.


Generated by Claude Code

Redirect-ledger liveness step 1 kept a record alive when any lock
inventory entry's resolved url merely CONTAINED the patch uuid, so a url
on any host spelling the uuid (or a Socket url for another patch whose
grant token is that uuid) kept a dead hosted record alive.

Step 1 now classifies the url with the one origin classifier,
`patch::redirect::hosted_patch_uuid`, over Socket's host and the run's
patch-server origins, and requires the classified uuid to equal the
record's. Discovery records the origins it was run with
(`Discovery::patch_server_origins`, set in `discover_with_ctx`), so
liveness classifies with exactly the origins discovery did; every
`LedgerLiveness` caller already holds a discovery built from the run's
`DiscoverOptions`. Socket-host and configured-origin urls for the
record's uuid stay live; anything else falls to step 2 (resolves
elsewhere: dead).

The uv script-lock regression test relied on the substring match for an
unconfigured internal host; it now shows that host dead without the
origin and live with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hosted mode pins a patched crate in the root Cargo.toml AND every
workspace-member manifest that declares it, and vendored mode's
live-redirect refusal reads member manifests too. Discovery read only
the root manifest, so a crate the root does not declare was "the lock is
the only word": a member whose declaration was reverted to crates.io
(no `registry = "socket-patch-<uuid>"`) left a ref attested although
cargo re-resolves that dependency from crates.io.

Discovery now enumerates the members the way the hosted rewriter does
(`cargo_workspace::member_manifests_in`, filtered by the rewriter's own
`is_cargo_member_manifest_key`, so nothing under `.socket/` or `target/`)
and applies its manifest-declaration check across root + members. Each
declaration carries its file: diagnostics, the conflict-pair Cargo.toml
ref and lockless `UnlockedPin`s name the member manifest that holds the
pin. `[patch]` stays root-only (cargo ignores a member's).

The member walk over a `DiskSnapshot` now notes every manifest it reads
and directory it probes or lists with the snapshot instead of asking for
its raw root, so a discovery run under a read-set recording (the re-scan
cache) keeps the recording usable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iveness

Cargo member walk (review 4, 6, 7):
- `cargo_workspace::member_manifests_in` is now async over the
  ProjectView: manifests are read with `read_text` (concurrently, bounded,
  ordered), globs list with `list_dir`, and directory probes use the view's
  own mediated `is_dir` / `is_symlink`. A snapshot's overlays and content
  cache apply and its recording notes the walk, so fresh manifests (a
  `.socket/` copy followed through `[patch]` included) no longer leave a
  racy read set. `DiskSnapshot::touch` / `touch_listing` are private again;
  `SnapshotTree` is gone. The in-memory engine keeps the sync walk
  (`member_manifests_in_memory`); disk-only callers keep `member_manifests`.
  A test pins the view walk to the disk walk over globs, `**`, excludes,
  path dependencies, `[patch]` copies, dot / `target` dirs and a
  symlinked member.

Discovery (review 5, 8):
- Members are walked only when they can matter: a lock with a
  Socket-sourced package, or no lock (then parsed only when some manifest
  carries a `socket-patch-` pin). The walk's texts are adopted into the
  ctx's read log and rule-11 recognition instead of being read again.
- The root `DocumentMut` is moved into the manifest list, not cloned.

Lockless pins (review 1, 3):
- `UnlockedPin` gains `excluded_reqs`: the requirements of the crate's
  unpinned declarations. The rewriter leaves a declaration alone only when
  its requirement excludes the patched version, so a member's
  `serde = "0.9"` beside a pinned `serde = "1"` no longer suppresses the
  pin; a ledger version must satisfy every pinned requirement and no
  excluded one. An unpinned declaration with no requirement still blocks.
- One pin and one diagnostic per (crate, patch): the union of the
  pinning declarations' requirements, every pinning manifest listed once.

Liveness (review 2):
- A lock entry whose url has the hosted shape for the record's uuid on an
  origin this run was not given is `RecordLiveness::Undecided`: never
  attested (`redirect_record_live` / `LedgerLiveness::redirect_record`
  stay attestation-grade) and never judged dead. `scan`'s takeover
  classifier and hosted-wiring-retained advisory use the new
  `LedgerLiveness::redirect_record_retained`, so a record written against
  a patch server a later run is not pointed at keeps its wiring. A url
  that merely contains the uuid (file name, query) is still dead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

foreign_hosted_uuid parsed the raw lock url, while hosted_patch_uuid
first unescapes \/, percent-decodes and strips a Cargo sparse+ /
registry+ prefix. A hosted url on an unconfigured origin that only
parses after that normalization was never Undecided and could be
marked Dead, dropping live wiring. Share the normalization. Addresses
one of the two Bugbot findings on this PR.

Co-Authored-By: Claude <noreply@anthropic.com>
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 94af3e4. Configure here.

@mikolalysenko
Mikola Lysenko (mikolalysenko) marked this pull request as ready for review November 6, 2026 12:12
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Burn-down agent: this needs an owner decision. Head 94af3e4f. CI is green apart from one PR-caused failure (coverage (2)) and the main-wide pdm/hishel e2e leg (with its ci-ok aggregate).

The failure. e2e_vex_lockfile yarn::uuid_on_a_foreign_host_is_not_a_hosted_reference (yarn.rs:1048–1085), for Classic with origin https://evil.example + a redirect ledger, not installed:

  • the test expects package_not_found
  • the PR returns redirect_unwired

Cause. The PR intends this change. Since 43fd5aa ("Match ledger liveness uuids by origin") and 650143e (RecordLiveness::Undecided), a lock entry whose hosted-shaped url is on an origin this run wasn't given is never attested. That contradicts what is documented today:

  • CLI_CONTRACT.md § "Patch hosts (manifest-less VEX)": "A pre-v5 redirect ledger still on disk can keep such a record in play under the pre-v5 rule: judged by the ledger's own recorded wiring, attesting only an installed tree that hashes to the record unless --no-verify…"
  • The e2e test's "DOCUMENTED FALLBACK" comment says the same. Its later cells expect hash_mismatch for a tampered tree and an attestation for a verified tree.

Question for the owner. Should a pre-v5 redirect-ledger record whose lock url sits on an unconfigured host (a staging server used without --patch-server-url, or a look-alike) still attest an installed tree that hashes to the record?

  • No (the PR as written). Next step: rewrite this e2e cell to expect redirect_unwired throughout, and update the CLI_CONTRACT paragraph.
  • Yes (the documented contract). Next step: vex must treat Undecided as "judged by the ledger, verify-only" instead of omitting it.

I didn't change either side, because either way the documented user-facing contract changes, and I won't rewrite a test's expectations just to get it green.

Minor, for whoever picks this up: in 94af3e4, hosted_patch_url_uuids's doc comment now sits above normalized_hosted_url (crates/socket-patch-core/src/patch/redirect/mod.rs ~7646). It needs splitting back.

Bugbot reviewed 94af3e4f.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants