Skip to content

conformance: what the unswept 42% of ruby/spec actually holds - #49

Merged
tannevaled merged 1 commit into
mainfrom
unswept-measured
Nov 10, 2026
Merged

tannevaled merged 1 commit into
mainfrom
unswept-measured

Conversation

@tannevaled

@tannevaled tannevaled commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

This page has always said library/, optional/, command_line/ and security/ are "not run at all" and that the figure is therefore not a percentage of Ruby. True — and it left the reader unable to tell whether the unmeasured half was a rounding error or the bulk of the work.

Measured once, against MRI 4.0.7 through the same shim:

spec files in the corpus 3 814 — 2 206 swept, 1 608 not
files where the two engines differ 970
MRI's score across those files 3 676
rbgo's score across the same files 1 335
files rbgo produces no result for 152 — 75 of them hang

Three readings matter more than the total, and each changes what somebody would do next:

  • 78 of the unswept files cannot be measured under this shim at all. All 32 command_line/ files are built on ruby_exe, and the 46 optional/capi/ files need a compiled C extension — MRI itself scores 0 in them. That share asks for harness work, not interpreter work.
  • The largest single debt is a hang, not a missing feature. Net::FTP is 442 examples behind and 39 of its 56 spec files are killed by the clock on rbgo, none on MRI; with Net::HTTP, Socket and Timeout that is 75 spec files (ruby#916). Until ruby#915 the harness called every one of them "did not load" — the opposite finding.
  • The rest is concentrated in eight modules, and the outright absences (Coverage 53, StringScanner's byte methods 27, CGI escapes 19, WeakRef 9) are small beside them.

It sits behind a warning callout saying it is one dated run and not a gate — nothing re-measures it, and the only machine-checked number on the page stays the gated language/ + core/ baseline.

⛔ The gate refused the first draft, and it was right

The row originally read

| examples MRI passes in them / rbgo passes | 3 676 / 1 335 |

which has exactly the shape of a conformance-total claim, so conformancefigure compared 3 676 against the gate's 23 672 and failed the build. Its own rule is to make the sentence unambiguous rather than widen the pattern, so the row is now "MRI's score across those files". The checked-claim count went 6 → 5 — the one false candidate leaving.

5 total claim(s), 2 version claim(s) and 6 stubbed-helper mention(s) checked
against the gate's 23672 at Ruby 4.0.9 (derived from RUBY_VERSION);
the shim stubs output_to_fd, ruby_cmd, ruby_exe

Refs go-embedded-ruby/ruby#916 — it stays open; this page only stops describing
those 75 files as unimplemented.

🤖 Generated with Claude Code

This page has always said `library/`, `optional/`, `command_line/` and
`security/` are "not run at all" and that the figure is therefore not a
percentage of Ruby. True, and it left the reader unable to tell whether
the unmeasured half was a rounding error or the bulk of the work.

Measured once, against MRI 4.0.7 through the same shim: 970 files differ,
MRI scores 3 676 examples across them and rbgo 1 335, and 152 files
produce no result on rbgo.

Three readings matter more than the total:

  - 78 of the unswept files CANNOT be measured under this shim at all.
    All 32 command_line/ files are built on ruby_exe and the 46
    optional/capi/ files need a compiled C extension, so MRI itself
    scores 0 in them. Harness work, not interpreter work.
  - The largest single debt is a HANG. Net::FTP is 442 examples behind
    and 39 of its 56 files are killed by the clock on rbgo, none on MRI;
    with Net::HTTP, Socket and Timeout that is 75 spec files (ruby#916).
    Until ruby#915 the harness called every one of them "did not load",
    which is the opposite finding.
  - The rest is concentrated in eight modules, and the outright absences
    (Coverage 53, StringScanner's byte methods 27, CGI escapes 19,
    WeakRef 9) are small beside them.

Marked as one dated run and NOT a gate, in a callout: nothing re-measures
it, and the only machine-checked number on the page stays the gated
language/ + core/ baseline.

⛔ conformancefigure refused the first draft, and it was right: a row
reading "examples MRI passes in them / rbgo passes | 3 676 / 1 335" has
exactly the shape of a conformance-total claim, and the tool compared
3 676 against the gate's 23 672. Its own rule is to make the sentence
unambiguous rather than widen the pattern, so the row is now "MRI's score
across those files". The claim count went 6 -> 5, which is the one false
candidate leaving.

Co-Authored-By: Claude Opus 5 <[email protected]>
@tannevaled
tannevaled merged commit bc1376d into main Nov 10, 2026
2 checks passed
@tannevaled
tannevaled deleted the unswept-measured branch November 10, 2026 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant