Skip to content

Say what the unswept trees hold, now that they have been measured #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tannevaled merged 1 commit into from
Nov 10, 2026
Merged

Say what the unswept trees hold, now that they have been measured #39

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions layouts/_default/list.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ <h3>Runtime conformance — ruby/spec</h3>
<ul>
<li>CI gates <b>file by file</b>, not on a total: no spec file may pass fewer examples than its recorded row, and none may stop loading. The sum of those rows is <b>23,672</b>, re-recorded at Ruby 4.0.9 (2026-11-03) on the pinned runner that enforces it — it is what CI guarantees, while the measured total above is what one dated run did. One file (<code>core/random/bytes_spec.rb</code>) is unstable between 8 and 9 passing examples; since 2026-11-04 its row holds <b>8</b>, its floor, because a row recorded at the high draw makes the next ordinary run a red build. This replaced a single shrink-only floor of 22,475, which needed a margin wider than the largest spec file and so could not see a regression smaller than that.</li>
<li><b>Only <code>language/</code> and <code>core/</code> are swept.</b> ruby/spec's <code>library/</code>, <code>optional/</code>, <code>command_line/</code> and <code>security/</code> trees are not run at all, so this figure is <b>not</b> a percentage of Ruby, and most of the standard library sits outside it.</li>
<li>Those unswept trees were measured <b>once</b>, on 2026-11-10, against MRI 4.0.7 through the same shim — a dated run, not a gate. Across the 970 files where the two engines differ the gap is <b>2,341</b> examples, and the shape matters more than the size: <b>78 files cannot be measured at all</b> (every <code>command_line/</code> spec is built on <code>ruby_exe</code>, and <code>optional/capi</code> needs a compiled C extension — MRI itself scores zero in them), the <b>largest single debt is a hang rather than a missing feature</b> (<code>Net::FTP</code> is 442 examples behind, with 39 of its 56 files killed by the clock on rbgo and none on MRI), and the outright absences are comparatively small (<code>Coverage</code> 53 examples, <code>WeakRef</code> 9).</li>
<li>Of the examples that ran, <b>93.3%</b> pass (22,488 of 24,116); 477 more are skipped because the shim stubs <code>ruby_exe</code>, <code>ruby_cmd</code> and <code>output_to_fd</code> — three helpers that would need a subprocess or real file-descriptor redirection — and 10 of 2,206 spec files produce no result. Skips and unreadable files sit outside the ratio entirely.</li>
<li>The guarantee has climbed from a floor of <b>6,000</b> on 2026-08-03 to a per-file baseline summing <b>23,672</b>. Two earlier jumps were fixes to the <b>measurement</b> — the mspec shim, and a parser upgrade that made 13 previously-unparseable spec files visible — which are gains in what the measurement can <i>see</i>, not in what the VM can <i>do</i>.</li>
</ul>
Expand Down
Loading