[panyam]

Draft for #451. On 2026-10-02 I offered to build it from the traceability run. This PR does that, so the design question now has code to look at.

A scenario can pass while most of its checks never run. In #451, AmirK-S showed a server that rejected every request and still passed input-required-result-validate-input 3/3. Pass/fail can't show this, but the number of checks that ran can.

The weekly traceability run already runs every scenario against the reference SDK. npm run traceability now also records which check IDs each scenario emitted, under a new scenarios key in traceability.json. Keys are client:<scenario> and server:<scenario>. Only SUCCESS, FAILURE and WARNING count. INFO and SKIPPED rows don't.

After each scenario, the runner compares the IDs it emitted against that scenario's baseline.

• It prints Coverage: 3/12 expected checks emitted. Not emitted: ... under the per-scenario results.
• The suite summary adds (coverage 3/12) to that scenario's line.
• It writes coverage.json next to checks.json for downstream tools.

Some runs get no baseline, so they can't show a false gap. The reference run used each scenario's default spec version, so a run at another version gets no baseline. A scenario missing from the manifest gets none either. In both cases the runner prints nothing about coverage.

Coverage never changes the exit code. You may want a short count to fail a run, but that's a policy call, so I left it to you.

Tests:

• src/coverage.test.ts covers two cases from Expose expected-vs-emitted check coverage per scenario #451's acceptance list, a thin run that passes and a setup failure. It also covers INFO and SKIPPED handling and the spec-version gate.
• src/traceability/index.test.ts covers mapping result dirs to scenario keys. One case is a client scenario whose name starts with server-.

I regenerated the manifest locally from sdk typescript-sdk@main --suite all, which gave 113 scenarios and 643 IDs. Then I ran server --suite all against two servers.

The first is a Go SDK test server that doesn't declare the skills extension. All three skills scenarios pass on one check, wire-schema-valid, and every other row is SKIPPED:

✓ sep-2640-skills-directory: 1 passed, 0 failed (coverage 1/7)
✓ sep-2640-skills-enumeration: 1 passed, 0 failed (coverage 1/32)
✓ sep-2640-skills-manifest: 1 passed, 0 failed (coverage 1/6)

The skips are intended. Without a count, though, the summary reads as three clean passes.

The second answers every request with -32000 Server not initialized, like the server in AmirK-S's comment. Six scenarios go green, including input-required-result-validate-input 3/3, and coverage flags none of them. The reference SDK emits the same 3 IDs there, so the count matches. Those cases still need the control calls in #537, and the count doesn't replace them.

This PR doesn't include traceability.json itself. The weekly workflow regenerates it and fills scenarios. Until then the runner prints nothing new.

Open questions:

• Should the baseline come from the reference SDK alone? If the reference SDK fails a scenario early, that scenario gets a thin baseline.
• Should schemaVersion move to 2? The new field is additive, and I don't know whether plan.mcp.io checks the version.

Once this merges we can delete our own per-scenario count snapshot.

[pkg-pr-new]

Open in StackBlitz

npx https://pkg.pr.new/@modelcontextprotocol/conformance@586

commit: 71e15ac