From 69bc566388d8a5573d34e69ef2de6c980e34f9d0 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Tue, 10 Nov 2026 15:24:02 +0100 Subject: [PATCH] conformance: what the unswept 42% of ruby/spec actually holds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This page has always said `library/`, `optional/`, `command_line/` and `security/` are "not run at all" and that the figure is therefore not a percentage of Ruby. True, and it left the reader unable to tell whether the unmeasured half was a rounding error or the bulk of the work. Measured once, against MRI 4.0.7 through the same shim: 970 files differ, MRI scores 3 676 examples across them and rbgo 1 335, and 152 files produce no result on rbgo. Three readings matter more than the total: - 78 of the unswept files CANNOT be measured under this shim at all. All 32 command_line/ files are built on ruby_exe and the 46 optional/capi/ files need a compiled C extension, so MRI itself scores 0 in them. Harness work, not interpreter work. - The largest single debt is a HANG. Net::FTP is 442 examples behind and 39 of its 56 files are killed by the clock on rbgo, none on MRI; with Net::HTTP, Socket and Timeout that is 75 spec files (ruby#916). Until ruby#915 the harness called every one of them "did not load", which is the opposite finding. - The rest is concentrated in eight modules, and the outright absences (Coverage 53, StringScanner's byte methods 27, CGI escapes 19, WeakRef 9) are small beside them. Marked as one dated run and NOT a gate, in a callout: nothing re-measures it, and the only machine-checked number on the page stays the gated language/ + core/ baseline. ⛔ conformancefigure refused the first draft, and it was right: a row reading "examples MRI passes in them / rbgo passes | 3 676 / 1 335" has exactly the shape of a conformance-total claim, and the tool compared 3 676 against the gate's 23 672. Its own rule is to make the sentence unambiguous rather than widen the pattern, so the row is now "MRI's score across those files". The claim count went 6 -> 5, which is the one false candidate leaving. Co-Authored-By: Claude Opus 5 --- content/en/conformance.md | 46 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/content/en/conformance.md b/content/en/conformance.md index 02427e4..cfe3880 100644 --- a/content/en/conformance.md +++ b/content/en/conformance.md @@ -245,6 +245,52 @@ An earlier revision of this page reported run-to-run spread from an intermittent crash in `core/module/autoload_spec.rb`. That did not reproduce during this measurement: five consecutive runs of that file returned identical counts, and it appeared in none of the three full sweeps. + +### What the unswept 42 % holds — measured once, 2026-11-10 {#the-unswept-42-percent} + +The four trees above are not gated. They have now been measured a single time +against **MRI 4.0.7** through the same shim, so the gap is a number instead of a +guess — `cmd/specdelta` for the comparison, `cmd/specroll` for the rollup, on +the pinned corpus at `c604cce1`. + +| | | +| --- | --- | +| spec files in the corpus | 3 814 — 2 206 swept, **1 608 not** | +| files where the two engines differ | 970 | +| MRI's score across those files | 3 676 | +| rbgo's score across the same files | 1 335 | +| the gap | 2 341 | +| files rbgo produces no result for | 152 — **75 of them hang** | + +Three readings matter more than the total, and each changes what somebody would +do next. + +**78 of the unswept files cannot be measured under this shim at all.** Every one +of the 32 `command_line/` files is built on `ruby_exe`, and the 46 +`optional/capi/` files need a compiled C extension — so **MRI itself scores 0** +in them. That share of the corpus asks for harness work, not interpreter work, +and no amount of conformance would move it. + +**The largest single debt is a hang, not a missing feature.** `Net::FTP` is 442 +examples behind, and 39 of its 56 spec files have to be killed by the clock on +rbgo while none are on MRI. Counting `Net::HTTP`, `Socket` and `Timeout`, 75 +spec files hang +([ruby#916](https://github.com/go-embedded-ruby/ruby/issues/916)) — and until +[ruby#915](https://github.com/go-embedded-ruby/ruby/pull/915) the harness +reported every one of them as *"did not load"*, which is the opposite finding. + +**The rest is concentrated.** `matrix` 233, `bigdecimal` 180, `stringscanner` +154, `date` + `datetime` 179, `zlib` 130, `stringio` 94, `openssl` 93. The +outright absences are small beside them: `Coverage` 53 examples, three +`StringScanner` byte methods 27, four `CGI` escape files 19, `WeakRef` 9. + +{{< callout type="warning" >}} +**One dated run, and not a gate** + +Nothing re-measures this. It is a map of where the work is, not a figure to +quote — the only machine-checked number on this page remains the gated +`language/` + `core/` baseline above. +{{< /callout >}} ### Not every gain is VM conformance {#not-every-gain-is-vm-conformance} Two of the recent jumps came from fixing the **measurement**, not the