Skip to content

Keep vendored artifacts a lock references through escaped paths - #2022

Open
Mikola Lysenko (mikolalysenko) wants to merge 3 commits into
mainfrom
fix/revert-keep-check-escaped-paths
Open

Mikola Lysenko (mikolalysenko) wants to merge 3 commits into
mainfrom
fix/revert-keep-check-escaped-paths

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

When a vendored revert or GC decides whether it can delete .socket/vendor/<eco>/<uuid>/, it checks whether any live wiring file still names that path. That check was a plain contains. An older composer.lock writes .socket\/vendor\/composer\/…, and a Windows checkout may use backslashes. Neither spelling matched, so the check returned "unreferenced" and the revert deleted an artifact the lock still installs from. #2017 handles only the composer BOM case through the stranded refusal; this PR fixes the matching itself for every format.

Fix

Every "does this project file still name this .socket/vendor/ path?" check now goes through one matcher, utils::path_spelling::PathNeedle. It decodes \uXXXX and XML-entity escapes, collapses every run of / and \ to a single /, and can ignore ASCII case. The lookup tries an exact contains first, then makes one KMP pass over the folded byte stream. The needle is folded once, and the scanned file is never copied.

The case rule depends on what the check decides:

  • Keep-vs-delete checks fold case (PathNeedle::keep / mentions_path). A false "referenced" only keeps an artifact.
  • Checks that refuse fold case only where the project's file system does. vendor::common::refusal_needle decides this with utils::fs::folds_case, which looks up .socket/ under a flipped-case name. On a case-sensitive disk, a differently-cased path is another directory and must not block a revert.

Discovery's identity sweep keeps fold_path_spellings, which is unchanged and moved into the util along with decode_escapes. It leaves separator runs alone because its hosted-url detection relies on ://. The golden corpus test still passes.

Every check moved onto the matcher

Check Decision Case
vendor::common::any_live_file_references composer / Maven / NuGet drift-keep any
vendor::nuget_feed::config_references keeps the NuGet lock pin any
vendor::npm_flavor::lock_text_mentions_uuid npm unwired guard; npm, yarn classic and yarn berry post-restore refusals disk
vendor::pypi::pypi_reference_clause (post-restore keep, dry-run preview) keep any
vendor::pypi::pypi_refusal_clause (unwired guard, vendor --check via pypi_reference) refuse / report disk
vendor::pypi::unwire_superseded residual check refuse re-vendor (the old uuid dir is swept as stale afterwards) disk
vendor::pypi::superseded_entry lifts a refusal into a supersede disk
vendor::pypi_lock::still_references_artifact, called from pypi-lock drift-keep any
vendor::pypi_lock::still_references_artifact, called from Hatch revert failure disk
vendor::pypi_requirements residual warning keep (vendor_revert_residual_reference) any
vendor::pypi_uv already-converged checks, the half-revert drift keep, revert_array_elements keep any
CLI scan_vendor_references_any_case (orphan sweep: vendor --revert, scan --prune, rollback, remove) keep any
CLI scan_vendor_references (vendor ledger-missing gate, rollback ledger-less gate, vendor --check, repair) refuse / report disk

Checked and left alone (no path text match, or not a deletion or refusal)

  • npm_flavor::uuid_proven_unreferenced matches on the uuid alone, case-insensitively, and parses the JSON.
  • pnpm_lock::pnpm_entry_in_use works on the parsed lock.
  • The pipenv relock-retire check matches the uuid alone.
  • composer_lock::stranded_wired_packages works on parsed JSON.
  • The CLI stale-artifact sweep decides from the ledger.
  • pypi_uv respell_unrecorded_path_entries compares parsed TOML values and only respells.
  • pnpm_lock file: copy advisory, yarn_classic_berry_migration_risk and pypi_requirements direct-reference detection only detect or warn.

Review findings

  1. unwire_superseded residual uses contains: confirmed and fixed. New test superseding_uuid_refuses_over_an_escaped_residual_reference fails on the old check: the uv flavor re-vendored, and the old uuid dir would then have been swept. It passes with the refusal needle.
  2. still_references_artifact uses contains: confirmed and fixed. New test leftover_artifact_reference_is_detected_in_any_spelling covers it. It takes the caller's needle: broad for pypi-lock, which keeps; disk-case for Hatch, which fails the revert.
  3. Orphan-sweep scan is case-sensitive: confirmed and fixed. The orphan sweep now reads scan_vendor_references_any_case. The refusing callers keep scan_vendor_references, which folds case only on a case-insensitive disk, so they don't refuse over a path that resolves elsewhere. Covered by test any_case_scan_recovers_a_recased_reference.
  4. Folds one escape at a time, so \\\\ / \\\/ became // and missed: confirmed and fixed by collapsing separator runs in PathNeedle and canonical_path_text, with new cases in the util and CLI tests. Discovery's own fold is not changed, for the :// reason above.
  5. Remaining plain-contains checks: fixed. Every site is in the table above.
  6. Perf: done. Exact fast path, single pass with no file copy, needle folded once per probe.
  7. Test doc placement: confirmed and fixed. The scan test now sits above the false-positive guard's doc comment, and it asserts that a different uuid's dir is never reported.
  8. Refusals over a case-insensitive match: confirmed and fixed with the disk-case refusal needle, and the stale "textual and EXACT" doc on guard_unwired_textual_revert is corrected. Test lock_text_mention_folds_case_only_where_the_disk_does expects whatever folds_case reports for the temp dir's .socket. That makes it a real failing-before test only on a case-sensitive disk such as Linux CI; on macOS it passes either way.

Tests

New since the first push:

  • superseding_uuid_refuses_over_an_escaped_residual_reference
  • leftover_artifact_reference_is_detected_in_any_spelling
  • lock_text_mention_folds_case_only_where_the_disk_does
  • CLI any_case_scan_recovers_a_recased_reference
  • path_spelling cases for separator runs, exact case, KMP overlap and canonical_path_text

Still in place from the first push: the revert, composer, npm and CLI scan escaped-spelling tests.

Runs:

  • core --lib vendor:: vex::discover:: utils::path_spelling: 3326 passed, golden test included.
  • CLI --lib repair, orphans and rollback tests: 78 passed.
  • mode_migration_composer: 7 passed.
  • mode_migration_npm: 32 passed.
  • mode_migration_pypi: 79 passed.
  • in_process_rollback_vendored: 23 passed.
  • in_process_rollback_all_ecosystems: 28 passed.
  • repair: 129 passed.
  • cargo clippy --locked --workspace --all-features -- -D warnings and cargo fmt --all -- --check are clean.

Review findings, round 2 (/code-review high on 5165d5c, cloud routine)

  1. Deletion gates matched case-sensitively on case-sensitive disks: confirmed and fixed. Round 1 made the "refusal" checks fold ASCII case only where the disk folds it. Several of them are also deletion gates: when they miss, the revert deletes the artifact. A committed lock is installed on macOS and Windows too, where .Socket/Vendor/... resolves to the artifact. So a Linux CI vendor --revert deleted, and committed the deletion of, an artifact those teammates still install from. Every check whose miss deletes now uses PathNeedle::keep (every spelling, any case):

    • the npm, yarn classic and yarn berry post-restore check (lock_text_mentions_uuid);
    • the unwired pypi revert guard;
    • the Hatch revert's leftover-reference check, which now matches the pylock revert's;
    • the superseded pypi entry's residual check before the old uuid dir is swept.

    refusal_needle remains only where a miss is safe: lifting another refusal (superseded_entry) and the vendor --check report. Tests: lock_text_mention_folds_case_on_every_disk (replaces lock_text_mention_folds_case_only_where_the_disk_does) and unwired_python_entry_revert_refuses_on_a_recased_reference. Both failed on 5165d5c.

    • Behavior change: on a case-sensitive disk, a project file naming a re-cased .socket/vendor/<eco>/<uuid>/ path now blocks the revert instead of letting it delete the artifact.
  2. The discovery identity-sweep comment was stale: fixed.

Declined:

  • The folds_case probe on non-unix systems with per-directory case sensitivity, and per-directory folding deeper in the path. After this fix those probes gate only safe-on-miss refusals and the CLI's refusal scan, so a wrong answer can no longer delete anything.
  • The KMP matcher and its pre-check. This is performance and simplification only. Lock texts are small next to the I/O.
  • Duplicated probe construction in the CLI. This is a refactor.

Tests (cloud run): cargo test -p socket-patch-core --lib -- vendor:: utils::path_spelling vex::discover:: gives 3323 passed. Its 2 failures are chmod write-failure tests that root ignores. -p socket-patch-cli --lib -- repair vendor and --test in_process_vendor pass. clippy and fmt are clean.


Note

Medium Risk
Changes deletion and revert refusal logic across many ecosystems; mistakes could delete artifacts still referenced by locks or block reverts incorrectly, though the design biases keep paths toward safety.

Overview
Replaces plain contains checks for “does this file still name .socket/vendor/...?” with a shared PathNeedle matcher in utils/path_spelling that decodes JSON/XML escapes, collapses / and \ runs, and optionally ignores ASCII case.

Keep-vs-delete paths (drift keep, orphan sweep, residual warnings) use PathNeedle::keep / scan_vendor_references_any_case so a false positive only retains an artifact. Refusal paths (vendor --check, unwired revert guards, rollback ledger gates) use refusal_needle, which folds case only when folds_case shows the project’s .socket/ tree is on a case-insensitive volume.

The CLI vendor-reference scan now runs canonical_path_text before slicing references; discovery reuses fold_path_spellings instead of a local escape decoder. Composer, npm, NuGet, PyPI (uv/requirements/Hatch/lock), and any_live_file_references are updated accordingly, with tests for escaped slashes, doubled backslashes, and re-cased anchors.

Reviewed by Cursor Bugbot for commit 5165d5c. Configure here.


Generated by Claude Code

The vendored revert's keep-check (`any_live_file_references`) did a plain
text match of `.socket/vendor/<eco>/<uuid>` against the live wiring files.
An older composer.lock writes `.socket\/vendor\/composer\/...`, a Windows
checkout may write single or doubled backslashes, and either spelling
missed the match: the check read "unreferenced" and the revert deleted an
artifact the lock still installs from.

Discovery's identity sweep already folds every one of those spellings
(`\uXXXX` / XML-entity decode, then `\/`, `\\` and `\` to `/`). Move that
fold into a pure util, `utils::path_spelling` (`fold_path_spellings`,
plus `mentions_path` for "does this text name this path"), have the sweep
call it unchanged, and route every vendored deletion gate that matched a
vendor path as raw text through it:

- `vendor::common::any_live_file_references` (composer / Maven / NuGet
  drift-keep in `revert::finish`)
- `vendor::npm_flavor::lock_text_mentions_uuid` (npm unwired and
  post-restore guards, yarn classic and berry post-restore guards)
- `vendor::pypi::pypi_reference_clause` (pypi unwired / post-restore keep)
- `vendor::nuget_feed::config_references` (keeps the lock pin while a
  config still routes at the feed)
- the CLI's `scan_vendor_references`, which the orphan sweep (`vendor
  --revert`, `scan --prune`, `rollback`, `remove`) uses to decide which
  un-ledgered dirs are still wired. It folded `\` to `/` only, so
  `\/` became `//` and the reference was lost.

`mentions_path` also matches ASCII case-insensitively. Every change only
widens what counts as a reference, so the failure mode is a kept
artifact, never a deleted one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mikolalysenko
Mikola Lysenko (mikolalysenko) marked this pull request as ready for review November 5, 2026 16:49
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 5, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Labeled Ready for review (burn-down agent).

  • Head: d4181c650ff7146bd2c275a753bffb2af6c67870
  • CI: 80/82 checks green; coverage (1) failed once with no log and passed on rerun (flake). The only red is the e2e_vex_build pdm leg (+ its ci-ok rollup), which is main-wide: hishel 2.0.0 breaks pdm lock on every PR. Fix pdm e2e red on every PR: pin hishel<2 #2028 fixes it; merge main in after it lands.
  • Bugbot: reviewed this head, no issues found.
  • Mergeable, no CHANGELOG.md change.
  • Slack announcement not sent this run (no Slack send tool available); the next run will retry.

Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Final review brief

What it does. Discovery already normalised path spellings (\uXXXX and XML-entity escapes, and \/, \\ and \ turned into /). This PR moves that fold, unchanged, into utils::path_spelling. Every vendored keep-or-delete gate that used a plain contains now calls mentions_path, which matches after the fold, ignoring ASCII case. As a result, revert and sweep no longer delete an artifact that an older \/-escaping composer.lock, or a backslashed Windows lock, still installs from.

Risk: low. The change can only add matches. The needles (.socket/vendor/<eco>/<uuid>) contain no \ or & and start with ., so an escape just before one can't swallow any of its characters, and anything that matched before still matches. Extra matches only ever lead to keep or refuse, never to a deletion. Discovery behaves the same: the fold is moved verbatim and the golden fixtures pass.

Look here

Verified

  • Read the full diff (10 files) and checked the callers in npm_lock, yarn_classic, yarn_berry, bun, nuget_feed, revert.rs and vendor --check.
  • cargo test -p socket-patch-core --lib, filtered to path_spelling, the new gate tests and the hosted golden fixtures: 18 passed. A broader filtered run: 59 passed. I didn't run the CLI repair test locally; CI runs it.
  • CHANGELOG.md is untouched, and there's no debug code left in or unrelated edits.
  • CI: ci-ok is red only on the known pdm e2e (e2e_vex_build pdm, hishel 2.x), which Fix pdm e2e red on every PR: pin hishel<2 #2028 fixes. clippy is green.

Nits (non-blocking, for the author)

  • repair.rs:1284:``` the new test was inserted between scan_ignores_non_vendor_socket_mentions's doc comment and that function, so the doc now sits on the wrong test.
  • decode_escapes became pub, but only its own module uses it. pub(crate) would be enough.
  • Theoretical, and shared with discovery: in a single-backslash Windows path, a segment that starts with u plus 4 hex digits is decoded as an escape. Real locks don't write paths like that.

Auto-merge is not armed: ci-ok has to go green first. Once it's green it gets armed, and approving then sends it straight to the merge queue.


Generated by Claude Code

Review fixes for #2022:

- One matcher, `utils::path_spelling::PathNeedle`: escapes decoded, every
  run of `/` and `\` collapsed to one `/` (so `\\\\` and `\\\/` match), an
  exact `contains` fast path, then a single KMP pass over the folded byte
  stream with the needle folded once. No copy of the scanned file.
- Keep-vs-delete checks fold ASCII case (`PathNeedle::keep`). Checks that
  REFUSE fold case only where the project's file system does
  (`vendor::common::refusal_needle`, probed via `utils::fs::folds_case`),
  so a differently-cased path on a case-sensitive disk no longer blocks
  `vendor --revert`.
- pypi: `unwire_superseded`'s residual check and `superseded_entry` read
  through the refusal needle (a `\/` or backslashed leftover was missed and
  the old uuid dir then swept as stale); `still_references_artifact`
  takes the caller's needle (broad for the pypi-lock drift keep, refusal
  for Hatch); `pypi_reference_clause` is broad for the post-restore keep
  and refusal-mode for the unwired guard and `vendor --check`; the
  requirements residual warning and the uv converged/drift checks use the
  broad needle.
- npm: `lock_text_mentions_uuid` uses the refusal needle; its
  "textual and EXACT" doc is corrected.
- CLI scan: `canonical_path_text` collapses separator runs; the orphan
  sweep reads `scan_vendor_references_any_case` (ASCII case-insensitive,
  like the revert keep-checks) while the refusing callers fold case only
  on a case-insensitive disk.
- Test doc placement fixed; a different-uuid assertion added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 5165d5c. Configure here.

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Ready for review re-confirmed at the current head (burn-down agent).

  • Head: 5165d5c4
  • CI: 47/49 green or skipped. The 2 red checks are main-wide: e2e (ubuntu-latest, e2e_vex_build, … pdm:: …) fails with hishel 2.x (module 'hishel' has no attribute 'SyncSqliteStorage'), the same failure as pdm e2e on main; ci-ok is red only because of that job. The fix is Fix pdm e2e red on every PR: pin hishel<2 #2028.
  • Bugbot: reviewed 5165d5c4 with no findings.
  • Mergeable: yes, no CHANGELOG change.
  • Slack: not announced; this run had no Slack send tool.

Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Final review brief (updated for 5165d5c)

What it does. The revert and orphan-sweep checks now spot a lock or config file that still names a .socket/vendor/... artifact in any spelling: JSON or XML escapes, \/, or any run of backslashes. That way an artifact a lock still installs from is never deleted. Checks that keep an artifact ignore case. Checks that refuse a revert ignore case only when the file system does (folds_case).

Risk: low-medium. The change mostly widens which files count as still referencing an artifact, so it errs toward keeping it. The one narrowing is on a case-sensitive disk, where a reference in a different case no longer blocks a revert. That is correct, because it names a different directory there. The diff is wide (12 files, many revert paths), which is why this isn't rated low.

Look here

  • path_spelling.rs:74: found_in / walk_folded, the KMP match over the folded byte stream
  • common.rs:467: refusal_needle
  • fs.rs:96: folds_case, the probe for whether the disk ignores case
  • pypi.rs:2836: the residual check in unwire_superseded (and superseded_entry at L2700)
  • repair.rs:70: parse_reference_folding / find_socket_anchor

Verified

  • I read the new commit and the full PR diff. The matcher can't index past the needle. It slices only at ASCII \\ or &, so it never splits a UTF-8 character. Lower-casing ASCII keeps byte offsets, so the recased leaf slicing in repair.rs is safe. No path built here becomes a delete target.
  • The new tests (superseding uuid, npm case) would fail without the change. CHANGELOG.md is untouched, with no CRLF, debug code or scope creep.
  • I couldn't run the tests locally this hour because the runner's disk was full, so this relies on CI. clippy is green. ci-ok is red only on the shared pdm e2e (e2e_vex_build pdm, hishel 2.x), which Fix pdm e2e red on every PR: pin hishel<2 #2028 fixes.

Nits (non-blocking)

  • %2F-style percent escapes are not decoded. That was already true before this PR.
  • On Windows, folds_case compares only file types, because there is no stable inode. A per-folder case-sensitive directory (WSL-style) could report the disk as ignoring case.

Auto-merge is not armed yet: ci-ok must go green first, after #2028 lands and this branch picks it up.


Generated by Claude Code

Second review round. Round 1 made the "refusal" checks fold ASCII case
only where the project's disk does. Several of them are also deletion
gates: a miss lets the revert delete the artifact. A committed lock or
config is installed on case-folding disks (macOS, Windows) too, where a
re-cased `.Socket/Vendor/...` path does resolve to the artifact, so a
case-sensitive CI run deleted (and committed the deletion of) an
artifact teammates still install from.

These now use `PathNeedle::keep` (every spelling, any case):
- the npm / yarn classic / yarn berry post-restore check
  (`lock_text_mentions_uuid`);
- the unwired pypi revert guard;
- the Hatch revert's leftover-reference check (now the same as the
  pylock revert's);
- the superseded pypi entry's residual check before the old uuid dir is
  swept.

`refusal_needle` remains only where a miss is safe (lifting another
refusal; the `vendor --check` report). The module docs and the
discovery identity-sweep comment say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants