Skip to content

Plan one lock model for every lockfile reader - #2001

Open
Mikola Lysenko (mikolalysenko) wants to merge 9 commits into
mainfrom
lock-model/0-design
Open

Mikola Lysenko (mikolalysenko) wants to merge 9 commits into
mainfrom
lock-model/0-design

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

First PR of the lock-model/* stack. No product code changes.

Why

Three groups of code read the same project files, each its own way:

  • Lockfile discovery (vex/discover/) finds Socket wiring. It isn't only for vex: scan, vendor, the hosted engine, the hosted unwind, rollout and ledger liveness all call it.
  • The lock inventory (vendor/lock_inventory/) lists every package a lockfile resolves.
  • The writers (patch/redirect/, hosted/, vendor/) parse the files again to plan their edits.

A survey of every format found this duplicated work:

  • two Socket-origin classifiers (one checks a host allowlist, one accepts any host);
  • three Python precedence rules that disagree;
  • second walks of package-lock, yarn, vlt and pnpm locks;
  • a second parse of uv, pylock and script locks;
  • five walks of Cargo.toml dependency tables;
  • four readers of Maven's checksums.sha256.

The inventory also has no NuGet, JVM, Hatch or Deno models.

What this PR adds

  • docs/design/lock-model-unification.md: the target design and the full stack.

    • One per-project model whose entries carry an Origin: Registry, Hosted, Vendored, Other or Rejected.
    • File selection by install tree: precedence within a tree, union across trees.
      • Every Python tool's lock is its own tree.
      • Deno stays supported while deprecated, and its reader is ported too.
    • NuGet and JVM inventory models are in scope. The doc covers what committed files can and can't give for each.
    • The owner decisions of 2026-11-04 and 2026-11-05 (see the Decisions table).
  • docs/design/lock-model-divergences.md: every question the readers and modes answer differently, with its resolution and the open PR that fixes it.

  • scripts/loc-budget.py with scripts/loc-budget.json: counts implementation lines (no comments, no items whose cfg implies test, no test-only modules) for two groups.

    Group Lines today
    lock-readers 30,462
    lock-writers 88,362

    Each later PR in the stack lowers these numbers in the same commit.

  • CI: lint-ecosystems runs the counter with --warn. It adds an annotation when a group is over budget but never fails, so bug fixes landing on main in these files aren't blocked while the stack is open (the doc's Size budget section explains why). ci.yml forbids step-level continue-on-error, which is why it's a separate mode.

  • Tests: scripts/tests/test_loc_budget.py (31 tests) covers comment and cfg(test) stripping, cfg predicates, strings, chars and lifetimes, test-only fields, variants, parameters and match arms, test-module declarations (#[path], inline modules, #[cfg(test)]), module resolution, file listing, and the check, warn, report and --files modes.

Stack

A PR that changes behavior lists each change, pins it with a test and shows its golden diffs (principle 4 in the doc).

PR Branch Change
0 lock-model/0-design (#2001) This PR
1 lock-model/1-origin (#2002) The top-level origin module (not part of formats/): discovery's classifier and the redirect's hosted-URL rule, moved verbatim
2a lock-model/2a-pnpm-readers (#2009) Pilot, readers: one pnpm entry walk; the env-lockfile document is its own tree
2b lock-model/2b-pnpm-planner (#2029) Pilot, planner: one pnpm rewire planner for both modes; shrinkwrap.yaml beside pnpm-lock.yaml is shadowed
3–9 npm, yarn, bun/vlt, deno, python, cargo/go, gem/composer One reader per format; second walks removed
10 lock-model/10-nuget NuGet records and inventory
11a lock-model/11a-jvm-readers One pom model, interpolator, checksums reader and Gradle index grammar; JVM inventory entries
11b lock-model/11b-maven-planner One Maven planner for both modes; hosted gains reactors and profiles
11c lock-model/11c-gradle-sbt-scala One Gradle and one sbt planner; hosted gains scala-cli
12 lock-model/12-install-trees Install-tree selection across formats
13 lock-model/13-cleanup Final deletions and budget

Review findings

From the code review of this PR. Each loc-budget.py bug got a test in scripts/tests/test_loc_budget.py that failed before the fix (13 failed, then all 27 pass). A later commit (4b1fc91, from Bugbot's review) also keeps cfg(test) crate name: Type fields and const generic parameters from swallowing code, so there are 31 tests now.

  1. Fixed (confirmed). A cfg(test) struct field, enum variant, parameter or match arm drove the item scan below its own depth and swallowed the code after it. An item now ends at a top-level , when it is a field, variant, parameter or arm (with <…> generics tracked so HashMap<A, B> stays whole), and always just before a bracket that closes its container.
  2. Fixed (confirmed). #[cfg(test)] use a::{…}; left its ; line counted (vendor/lock_inventory/mod.rs, one line; vendor/npm_lock.rs, one line). A ; right after the closing brace now belongs to the item.
  3. Fixed (confirmed). cfg(all(unix, test)) and #[cfg(test)] were missed. cfg predicates are now parsed: test, an all(…) with such an argument, or an any(…) whose every argument is one. not(test) never counts, so "contains test anywhere" was narrowed to that. A file whose #[cfg(test)] sits outside any block is test-only, and so are its submodules.
  4. Fixed (confirmed), both parts. #[path] is now read from any attribute around the item, before or after #[cfg(test)], and an out-of-line #[path] resolves against the declaring file's own directory, as rustc does (inside inline modules, against their directory).
  5. Fixed (confirmed). A mod x; inside a test-only inline module is recorded with its inline path. This was live: vex/discover/testing/golden.rs (435 lines, declared inside #[cfg(test)] mod testing { mod golden; }) counted as implementation.
  6. Fixed. The Decisions table now records the narrowed repository-root rule (workspace runs only), and G1, J13 and P4 match it (see "Also" below).
  7. Fixed. This description's stack table said formats::origin and had no 2a/2b or 11a/b/c split; it now matches the doc, which also gained the 2a/2b split and PR numbers.
  8. Fixed. Four checksums.sha256 readers everywhere (doc problem table, J9, this description).
  9. Fixed (confirmed). git ls-files -z, split on NUL. The counter now reads only the group files, their siblings and each ancestor directory's module file (2.9 s to 2.0 s locally); the summary line is now all groups (the union of the groups) instead of the whole repository.
  10. Fixed. Warn-only stays and the doc's Size budget section says why. --files with an unknown group is now a usage error (exit 2) that lists the groups.

Budget after the counter fixes: lock-readers 30,898 to 30,462 (golden.rs and one };), lock-writers 88,363 to 88,362, lowered with --lower.

Also (owner decisions, 2026-11-05), now in the Decisions table:

The divergences rows G1, J9, J13, P4, P9, X12, N1, N2, N4, N5, N11, N12 and N14 now name the open PR that fixes them, and the fixes-on-main table has a PR column.

Testing

  • python3 -B -m unittest discover -s scripts/tests: 411 tests, OK (1 skipped).
  • scripts/loc-budget.py --check exits 0, with both groups at their budget.

Note

Low Risk
Documentation and auxiliary Python tooling only; CI uses a non-blocking warn mode so normal merges are unaffected.

Overview
Design-only kickoff for the lock-model/* stack: no changes to Rust product code.

Adds docs/design/lock-model-unification.md (single lock model, Origin, install trees, principles, owner decisions, JVM/NuGet scope, staged stack) and docs/design/lock-model-divergences.md (matrix of reader/mode disagreements and resolutions).

Introduces scripts/loc-budget.py with scripts/loc-budget.json: counts non-comment, non-cfg(test)-gated implementation lines for lock-readers and lock-writers groups; supports --check, --warn, --lower, and --files. scripts/tests/test_loc_budget.py exercises the counter and modes.

CI: the lint-ecosystems job runs loc-budget.py --warn, emitting GitHub annotations for over-budget groups without failing the job.

Reviewed by Cursor Bugbot for commit d6ccde3. Configure here.


Generated by Claude Code

Lockfile discovery (vex/discover), the lock inventory
(vendor/lock_inventory) and the hosted and vendored writers each read the
same project files their own way: two Socket-origin classifiers, three
Python precedence rules, second walks of package-lock, yarn, vlt and pnpm
locks, a second parse of uv/pylock/script locks, five Cargo.toml
dependency-table walks, three readers of Maven's checksums.sha256. The
inventory also has no NuGet, JVM, Hatch or Deno models.

docs/design/lock-model-unification.md describes the target, one
per-project model whose entries carry an Origin (Registry, Hosted,
Vendored, Other, Rejected), file selection by install tree (precedence
within a tree, union across trees), and the stack of PRs that gets there.

scripts/loc-budget.py counts implementation lines (no comments, no
cfg(test) items, no test-only modules) for two groups in
scripts/loc-budget.json: lock-readers (30,898 lines today) and
lock-writers (88,363). The lint-ecosystems job runs it with --warn, which
annotates a group over budget but never fails, so bug fixes in these files
are not blocked; each PR in the stack lowers the budget as it deletes
code.

No product code changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
formats/ models are guarded to stay free of I/O and of the host policy
(formats/mod.rs architecture_tests bans hosted_patch_uuid there), so the
classifier is a top-level `origin` module and a project layer applies it.
Spell out the three layers and where each part of the classifier moves
from.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decisions: inconsistencies between discovery, the inventory and the
two modes are bugs to remove, not behavior to preserve; hosted and
vendored share one core and differ only in where a rewired entry points;
when in doubt the hosted mode's behavior wins, and every such call is
listed in the doc so it can be overruled. Behavior changes are allowed
per PR, listed, pinned by tests and shown as golden diffs.

First decision recorded: pnpm 11's env-lockfile document counts as its
own install tree, as the hosted planner already treats it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/design/lock-model-divergences.md lists every question on which
discovery, the inventory, hosted and vendored mode answer differently
today, across all ecosystems, with the resolution under the principles
and flags for false attestations and target-specific differences.

Owner decisions recorded: the modes are ground truth over the readers;
vendored mode's Maven profile and reactor support and its scala-cli wiring
are back-ported to hosted as the planners merge (the JVM PR splits into
readers, Maven planner, and Gradle/sbt/scala-cli). One call is pending:
whether hosted keeps rewriting a shrinkwrap.yaml that pnpm never reads.

Corrects the install-tree section: npm-shrinkwrap.json does not shadow
package-lock.json; every consumer but the inventory already reads both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decisions: a dependency the user patched themselves is never
patched or attested by default, in every mode and package manager, unless
the user explicitly overrides; and no reader or writer reads project files
outside the git repository root.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #2001 found the counter miscounting in several shapes, each
now pinned by a test in scripts/tests/test_loc_budget.py:

- a cfg(test) struct field, enum variant, parameter or match arm drove
  the scan below the item's depth and swallowed the code after it; the
  item now ends at a top-level `,` or before the bracket closing its
  container
- `#[cfg(test)] use a::{..};` left its trailing `;` line counted
- cfg predicates are parsed: `all(unix, test)` and nested `all` count,
  `not(test)` doesn't, and a file opening with `#![cfg(test)]` is
  test-only
- `#[path]` is found among any attributes around the item and resolves
  against the declaring file's directory, as rustc does
- `mod x;` inside a test-only inline module (vex::discover's
  `testing { mod golden; }`) marks that file test-only
- `git ls-files -z` keeps paths with spaces whole, and only the group
  files plus their module declarers are read
- `--files` with an unknown group is a usage error, not a KeyError

golden.rs (435 lines) and two `};` lines leave the counts, so the
budget drops to 30462 / 88362.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #2001: the Decisions table said every read is bounded to the
repository root, while G1, J13 and P4 still said to read an ancestor
go.work, parent NuGet configs and every include. The owner narrowed
the rule on 2026-11-05, and the table now records it with the rest of
that day's calls:

- the bound applies to workspace runs only; --global / -g /
  --global-prefix runs are unbounded, and without a git checkout upward
  lookups behave as before
- a marker file above the repository root (below home) makes hosted and
  vendored refuse with a message naming it
- out-of-repository -r includes stay bounded in workspace runs
- N11: shrinkwrap.yaml beside pnpm-lock.yaml is shadowed, unless the
  project pins pnpm 1 / 2, where it governs and vendored refuses
- discovery withholds pnpm root and member-lock attestations whenever a
  pnpm-lock.<branch>.yaml exists
- user patches: the override is --allow-patch-conflicts, and vex never
  attests under a user patch

G1, J9, J13, P4, P9, X12, N1, N2, N4, N5, N11, N12 and N14 name the open
PRs that fix them, and the fixes-on-main table gains a PR column. The
problem table now counts four checksums.sha256 readers, as J9 does. The
stack table splits PR 2 into 2a / 2b with their numbers, and the size
budget section says why CI only warns and gives the corrected counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread scripts/loc-budget.py Resolved
…g code

loc-budget.py only recognized `pub name: Type` as a field, so a
#[cfg(test)] on a `crate name: Type` field or a `const N: usize`
generic parameter was read as an item. _item_end then didn't stop at the
comma and excluded the rest of the struct or parameter list from the
implementation count.

Treat `crate` followed by a name like `pub`, and read `const` as a
parameter when the attribute follows `<` or `,`, where no item can
start. A `crate::path` statement and a plain const item keep their
current handling. Reported by Bugbot on #2001.

Co-Authored-By: Claude <noreply@anthropic.com>
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@mikolalysenko
Mikola Lysenko (mikolalysenko) marked this pull request as ready for review November 6, 2026 12:12
@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 6, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Ready for review at 4b1fc91.

  • CI: 31/33 non-skipped checks green. The only failures are the main-wide e2e … pdm:: --ignored leg (module 'hishel' has no attribute 'SyncSqliteStorage', also red on main; Fix pdm e2e red on every PR: pin hishel<2 #2028 fixes it) and the ci-ok aggregate that follows from it.
  • Bugbot: reviewed 4b1fc91; no unresolved findings.
  • Promoted from draft by the burn-down agent this run. No Slack announcement was sent this run (the Slack send action wasn't available), so there's no slack-announced marker yet.

Generated by Claude Code

Follow-up PRs grep for the names this doc cites. OriginPolicy::hosted_uuid
exists nowhere in the tree; the function is DiscoverCtx::hosted_uuid (its
memoized fast path is the hosted_hosts cell). The npm flavor router lives
at vendor/lock_inventory/npm_family.rs, since doc paths are relative to src/.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011f9aeNYq2GHyrsNnPBepcC
@mikolalysenko Mikola Lysenko (mikolalysenko) removed the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 6, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[final reviewer] I reviewed this design PR. Nothing blocks it. I pushed one small fix in d6ccde3, so the brief will follow on that head:

  • docs/design/lock-model-unification.md: OriginPolicy::hosted_uuid does not exist in the tree, so I changed it to DiscoverCtx::hosted_uuid. I also fixed the router path to vendor/lock_inventory/npm_family.rs. Follow-up agents will grep for these names.

I removed Ready for review; the burn-down re-labels it once CI is green.

Non-blocking notes for the approver:

  • The description's test counts are stale. test_loc_budget.py has 31 tests, and scripts/tests runs 415 with 6 skipped. Both --check and --warn exit 0 at 30,462 / 88,362.
  • The lock-writers budget group covers all of vendor/, hosted/ and patch/redirect/, about 88k lines. Unrelated growth there will hide the stack's deletions.
  • The Decisions table (lines 77-85) commits the stack to these product calls, worth approving deliberately:
    • reads bounded to the git root;
    • every Python lock read as its own tree;
    • no hosted URLs on arbitrary hosts;
    • user patches never patched or attested, even with --allow-patch-conflicts;
    • NuGet/JVM in scope.

bugbot run


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit d6ccde3. Configure here.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 6, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Ready for review at d6ccde3e.


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Final review brief

What it does. First PR of the lock-model/* stack; no product code changes. It adds the design doc for merging the three lockfile-reading paths (discovery, lock inventory, writers) into one per-project model, a catalog of where readers disagree, and a small Python counter (scripts/loc-budget.py) that tracks implementation lines in the reader and writer groups so later stack PRs can show they delete code.

Risk: low. Docs plus a CI step that runs in --warn mode (annotates when over budget, always exits 0), so it can't block other PRs. The script only reads git ls-files output; no network, no writes outside --lower.

Look here

Verified

  • Ran python3 -B -m unittest discover -s scripts/tests on the head: 415 tests OK (6 skipped).
  • Ran scripts/loc-budget.py and --warn: both groups exactly at budget, exit 0.
  • CHANGELOG.md untouched; no unresolved review threads; Bugbot reviewed d6ccde3e with no findings.
  • CI: 30 green, 19 skipped, 2 red. The red ones are the main-wide e2e … pdm:: --ignored leg (hishel 2 removed SyncSqliteStorage) and the ci-ok aggregate behind it. Fix pdm e2e red on every PR: pin hishel<2 #2028 fixes that; it's not caused by this PR.

Changes I made. In the previous pass, d6ccde3: corrected two symbol/path names in the design doc (DiscoverCtx::hosted_uuid, vendor/lock_inventory/npm_family.rs).

Open questions. None blocking. Minor: the exclude entry vendor/test_support has no trailing slash, so it is a prefix match; harmless today.

Auto-merge isn't armed yet: ci-ok must go green first (after #2028 lands and this PR's CI re-runs). Then approving sends it straight to the merge queue.


Generated by Claude Code

Mikola Lysenko (mikolalysenko) added a commit that referenced this pull request Nov 6, 2026
Brings in #2002's review fixes (OriginPolicy imported from crate::origin,
the redirect re-export dropped, the shared host matcher, the stricter
architecture guards) and #2001's later counter fixes.

Restores crates/socket-patch-core/src/origin.rs to the lock-readers
group, which the previous merge (1d27a20) inherited the loss of from
1-origin's 4242ebc. Budget at this branch's counts: lock-readers
30767, lock-writers 88315.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mikola Lysenko (mikolalysenko) added a commit that referenced this pull request Nov 6, 2026
Brings in #2002's review fixes and #2001's later counter fixes through
2a. Restores crates/socket-patch-core/src/origin.rs to the lock-readers
group (lost in the earlier 1-origin merge). Budget at this branch's
counts: lock-readers 30766, lock-writers 88114, both below 2a's 30767
and 88315.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants