From 0d7a97088cea8ee226fb93b17ea7b5f5e943c420 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Mon, 9 Nov 2026 13:27:12 +0100 Subject: [PATCH] conformance: derive the stubbed-helper list from the shim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit conformance.md said the shim stubs "a few MSpec matchers (`complain`, `output`, `ruby_exe`, …)". Read against spec_helper.rb, two of the three named are implemented and the list was missing one: stubbed ruby_exe (:882), ruby_cmd (:883), output_to_fd (:280) NOT complain (:249), output (:279), be_computed_by (:199) The error runs in the direction that hides work. An example using a genuinely stubbed helper SKIPS; one using `complain` RUNS, and when the behaviour is missing it FAILS. A reader sent to look for it among the skips does not find it there -- ruby#909 was nearly missed exactly so. ruby#911 corrected the same sentence in the ruby repository and this page kept the wrong one, which is why this adds a check rather than another corrected paragraph. tools/conformancefigure now reads every `def … raise SpecSkip` out of spec_helper.rb -- the file it already fetches for CUR_VERSION -- and judges the pages against that set in both directions: a helper called stubbed that is not, and a stubbed helper the site names nowhere. Three things it does on purpose: - It is a VOCABULARY with its own self-test, like the total's and the version's. "Stub" is an ordinary word here: webassembly.md has a "graceful stub" row listing `sqlite3`, `sequel`, `activerecord`, and the paragraph recording this very correction says "an example using a stubbed helper skips; one using `complain` runs". Matched on the word alone, both were read as claims and this check failed conformance.md on its own explanation. Measured, then narrowed to three phrasings, with those two lines as negative cases. - The unreadable-shim refusal is HELD until the findings are printed. The file's own note says a structural complaint must never arrive before a concrete disagreement; written the obvious way, a missing stub list swallowed every stale-total finding in the suite. - `skip` is outside the set and the comment says so: it raises SkippedSpecError because the spec asked to be skipped, which is mspec behaviour rbgo implements, not a hole in the shim. Also: the unswept trees are `library/`, `optional/`, `command_line/` and `security/`. `command_line/` -- 32 spec files -- was missing from that list on this page, on the landing page and in the ruby README. Co-Authored-By: Claude Opus 5 --- content/en/conformance.md | 27 ++-- tools/conformancefigure/main.go | 218 ++++++++++++++++++++++++++- tools/conformancefigure/main_test.go | 143 +++++++++++++++++- 3 files changed, 373 insertions(+), 15 deletions(-) diff --git a/content/en/conformance.md b/content/en/conformance.md index 9849b87..02427e4 100644 --- a/content/en/conformance.md +++ b/content/en/conformance.md @@ -145,9 +145,17 @@ margin could not distinguish *one file failing to load* from *hundreds of specs regressing*. A run that gains examples overall still fails here if any single file drops, which a total could not see. -The shim stubs a few MSpec matchers (`complain`, `output`, `ruby_exe`, …), so -their examples count as *skipped* rather than passing — the record is a -deliberately **conservative lower bound**. The +The shim stubs `ruby_exe`, `ruby_cmd` and `output_to_fd` — three helpers that +would need a subprocess or real file-descriptor redirection — so their examples +count as *skipped* rather than passing, and the record is a deliberately +**conservative lower bound**. + +`complain`, `output` and `be_computed_by` are **implemented**, not stubbed; this +page said otherwise until 2026-11-09, and the error ran in the direction that +matters. An example using a stubbed helper *skips*; one using `complain` **runs** +and, when the behaviour is missing, **fails** — so a reader trusting the old +sentence would have looked for it among the skips, where it is not. That list is +now derived from the shim by `tools/conformancefigure` rather than retyped. The [`rubyspec-ratchet`](https://github.com/go-embedded-ruby/ruby/blob/main/.github/workflows/rubyspec-ratchet.yml) workflow enforces it on every push and pull request. @@ -220,12 +228,13 @@ a checked fact carry an unchecked one. It is **not** "rbgo implements 93 % of Ruby", and it cannot be turned into a percentage of the language: -- **Only `language/` and `core/` are swept.** ruby/spec's `library/`, - `optional/` and `security/` trees are **not run at all** — most of what people - mean by "the standard library" is outside this measurement entirely. -- **The shim is not mspec.** Some matchers are stubbed, so the examples needing - them are counted as *skipped*, never as passing. The total is a conservative - lower bound. +- **Only `language/` and `core/` are swept.** `run.sh` walks those two trees and + no others, so ruby/spec's `library/`, `optional/`, `command_line/` and + `security/` trees are **not run at all** — most of what people mean by "the + standard library" is outside this measurement entirely. +- **The shim is not mspec.** Three helpers are stubbed — `ruby_exe`, `ruby_cmd` and `output_to_fd` — + so the examples needing them are counted as *skipped*, never as passing. The + total is a conservative lower bound. - **Skips are outside the ratio.** The 93.3 % divides passes by the examples that ran; the 477 skips and the 10 unreadable files are in neither column. - **Examples are not weighted.** A heavily specified method contributes far more diff --git a/tools/conformancefigure/main.go b/tools/conformancefigure/main.go index abdadcf..93678c3 100644 --- a/tools/conformancefigure/main.go +++ b/tools/conformancefigure/main.go @@ -33,6 +33,7 @@ import ( "os" "path/filepath" "regexp" + "sort" "strconv" "strings" "time" @@ -90,6 +91,24 @@ var ( // recognise when there is only one copy left and say so, not treat the // absence of the second copy as an unreadable answer. shimDerivedRe = regexp.MustCompile(`(?m)^CUR_VERSION\s*=\s*RUBY_VERSION\s*(?:#.*)?$`) + + // ⛔ THE THIRD FACT THESE PAGES RESTATE, and the one that was wrong on both + // of them. The shim stubs some mspec helpers by raising SpecSkip; those + // examples are counted as SKIPPED, which is why every page says the total is + // a conservative lower bound. Which helpers those are was retyped, and on + // 2026-11-09 conformance.md said "`complain`, `output`, `ruby_exe`, …" when + // `complain` and `output` are implemented and `ruby_cmd` -- stubbed one line + // below the `ruby_exe` that was named -- appeared nowhere. + // + // The error runs in the direction that hides work: an example using a + // genuinely stubbed helper SKIPS, but one using `complain` RUNS and, when the + // behaviour is missing, FAILS. A reader sent to look for it among the skips + // does not find it there (ruby#909 was nearly missed exactly so). + // + // `skip` is deliberately outside this set: it raises SkippedSpecError, which + // this pattern does not match, because the spec ASKED to be skipped -- mspec + // behaviour rbgo implements, not a hole in the shim. + shimStubRe = regexp.MustCompile(`(?m)^\s*def\s+([a-z_][a-z0-9_]*[?!]?)\b[^\n]*\braise[\s(]+SpecSkip\b`) ) // Two shapes of CURRENT claim, because two sites phrase it two ways round. @@ -230,6 +249,108 @@ var vocabularyCases = []struct { {`
  • Of the examples that ran, 93.3% pass (22,488 of 24,116); 477 more are skipped because our shim stubs the matcher they need, and 10 of 2,206 spec files produce no result.
    • `, false, ""}, } +// The stubbed-helper vocabulary. +// +// A line makes this claim in one of three phrasings -- "the shim stubs X", +// "N helpers are stubbed: X", "the matchers stubbed are X" -- and the +// enumeration must sit on the same line as the phrase that introduces it. +// +// It is a VOCABULARY, like the total's, and for the same reason: "stub" is an +// ordinary English word on these sites. `webassembly.md` has a row about a +// "graceful stub" listing `sqlite3`, `sequel`, `activerecord` in code spans, +// and the paragraph recording this very correction says "an example using a +// stubbed helper *skips*; one using `complain` **runs**". Matched on the word +// alone, both would be read as claims about mspec helpers and both would fail a +// page that is right. Measured: they did, on the first run of this check. +// +// The cost of a vocabulary is the same here as there -- a phrasing missing from +// it is a claim the gate cannot refuse -- which is why the enumeration is kept +// to one line in both pages and why stubCases holds the real sentences. +// +// A line that DENIES stubbing is not a claim of it: "`complain`, `output` and +// `be_computed_by` are **implemented**, not stubbed" is the sentence that +// records the correction, and it names three helpers that must not be in the +// set. It is excluded by its negation rather than by its names, so the next such +// sentence is handled without touching this code. +var ( + stubClaim = regexp.MustCompile(`(?i)\bshim stubs\b|\b(?:helpers?|matchers?)\s+(?:are|is)\s+stubbed\b|\b(?:helpers?|matchers?)\s+stubbed\s+(?:are|is)\b`) + stubNegation = regexp.MustCompile(`(?i)\bnot stubbed\b|\bimplemented\b`) + // Markdown backticks for the docs, for the landing page's layouts. + // `language/` and `core/` carry a slash and so are never identifiers. + codeSpanName = regexp.MustCompile("(?:`|)([a-z_][a-z0-9_]*[?!]?)(?:`|)") +) + +// stubCases is this vocabulary's own control, in the same spirit as +// vocabularyCases and versionCases, and run on every invocation for the same +// reason: a pattern that matches nothing is indistinguishable from a site that +// states the list correctly. +var stubCases = []struct { + line string + want []string +}{ + // The docs' prose and its bullet, as this change writes them: the phrase and + // the whole enumeration on one line. + {"The shim stubs `ruby_exe`, `ruby_cmd` and `output_to_fd` — three helpers that", + []string{"ruby_exe", "ruby_cmd", "output_to_fd"}}, + {"- **The shim is not mspec.** Three helpers are stubbed — `ruby_exe`, `ruby_cmd` and `output_to_fd` —", + []string{"ruby_exe", "ruby_cmd", "output_to_fd"}}, + // The landing page's HTML, where a list item is one long line anyway. + {"
  • 477 more are skipped because the shim stubs ruby_exe, ruby_cmd and output_to_fd.
    • ", + []string{"ruby_exe", "ruby_cmd", "output_to_fd"}}, + // ⛔ The sentence that was WRONG, as it stood on 2026-11-09. It must be a + // claim, or the gate could not have caught it. + {"The shim stubs a few MSpec matchers (`complain`, `output`, `ruby_exe`, …), so", + []string{"complain", "output", "ruby_exe"}}, + // The correction itself: a denial, not a claim. + {"`complain`, `output` and `be_computed_by` are **implemented**, not stubbed; this", nil}, + // ⛔ THE SENTENCE THAT EXPLAINS THE CORRECTION, which names `complain` and is + // not a claim about what is stubbed. Matched on the word "stub" alone it was + // read as one, and this check failed conformance.md on its own explanation. + {"matters. An example using a stubbed helper *skips*; one using `complain` **runs**", nil}, + // Vague but true, and naming nobody: nothing to check. + {"- **The shim is not mspec.** Some matchers are stubbed, so the examples needing", nil}, + {"
  • 477 more are skipped because our shim stubs the matcher they need
    • ", nil}, + // ⛔ Real lines from this site that use "stub" about something else. The + // first names four gems and would fail a correct page. + {"| **graceful stub** | **succeeds** | the module loads, but constructing a connection / handle raises | `sqlite3`, `sequel`, `activerecord` |", nil}, + {" - *RPC, serialization & columnar data* — **`grpc`** (server + client stub,", nil}, + // A line about the suites, which names code spans and no stub. + {"- **Only `language/` and `core/` are swept.** ruby/spec's `library/`,", nil}, +} + +// matchStubs returns the helpers a line claims the shim stubs. +func matchStubs(line string) []string { + if !stubClaim.MatchString(line) { + return nil + } + if stubNegation.MatchString(line) { + return nil + } + var out []string + for _, m := range codeSpanName.FindAllStringSubmatch(line, -1) { + out = append(out, m[1]) + } + return out +} + +// checkStubVocabulary runs stubCases, like checkVocabulary, before any verdict. +func checkStubVocabulary() error { + for _, c := range stubCases { + got := matchStubs(c.line) + if len(got) != len(c.want) { + return &exitError{2, fmt.Sprintf("the stubbed-helper vocabulary is wrong: %q yielded %v, expected %v", + c.line, got, c.want)} + } + for i := range got { + if got[i] != c.want[i] { + return &exitError{2, fmt.Sprintf("the stubbed-helper vocabulary is wrong: %q yielded %v, expected %v", + c.line, got, c.want)} + } + } + } + return nil +} + // match returns the figure a line claims as the CURRENT total, if any. func match(line string) (string, bool) { if datedLine.MatchString(line) { @@ -403,6 +524,9 @@ func check(stdout, stderr io.Writer, root, baseline, runtimePath, shimPath strin if err := checkVocabulary(); err != nil { return err } + if err := checkStubVocabulary(); err != nil { + return err + } want, err := authoritative(baseline, timeout) if err != nil { @@ -416,6 +540,17 @@ func check(stdout, stderr io.Writer, root, baseline, runtimePath, shimPath strin return &exitError{2, fmt.Sprintf("cannot read the version the gate measures at: %v", err)} } + // ⛔ NOT a refusal here, and the ⛔ further down says why: a structural + // complaint must never arrive before a concrete disagreement. An unreadable + // stub list does not stop this tool reading a stale TOTAL, so the refusal is + // held until the findings have been printed. + wantStubs, stubErr := authoritativeStubs(shimPath, timeout) + isStub := map[string]bool{} + for _, s := range wantStubs { + isStub[s] = true + } + namedSomewhere := map[string]bool{} + type finding struct { where string got int @@ -425,9 +560,14 @@ func check(stdout, stderr io.Writer, root, baseline, runtimePath, shimPath strin where string got string } + type stubFinding struct { + where string + got string + } var wrong []finding var wrongVersion []versionFinding - checked, versionsChecked := 0, 0 + var wrongStub []stubFinding + checked, versionsChecked, stubsChecked := 0, 0, 0 err = filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error { if err != nil { @@ -476,6 +616,19 @@ func check(stdout, stderr io.Writer, root, baseline, runtimePath, shimPath strin }) } } + for _, name := range matchStubs(text) { + if stubErr != nil { + break // no oracle for this fact; the refusal waits its turn below + } + stubsChecked++ + namedSomewhere[name] = true + if !isStub[name] { + wrongStub = append(wrongStub, stubFinding{ + where: fmt.Sprintf("%s:%d", path, line), + got: name, + }) + } + } own := allVersions(text) ownHad := len(own) > 0 versions := append(own, versionsAcross(prev, prevHadVersion, text, ownHad)...) @@ -498,8 +651,9 @@ func check(stdout, stderr io.Writer, root, baseline, runtimePath, shimPath strin // "N of M, never N alone": a walk that matched nothing reports no // disagreement, which is the same output as a correct site. - fmt.Fprintf(stdout, "%d total claim(s) and %d version claim(s) checked against the gate's %d at Ruby %s (%s)\n", - checked, versionsChecked, want, wantVersion, howSpelled) + fmt.Fprintf(stdout, "%d total claim(s), %d version claim(s) and %d stubbed-helper mention(s) checked "+ + "against the gate's %d at Ruby %s (%s); the shim stubs %s\n", + checked, versionsChecked, stubsChecked, want, wantVersion, howSpelled, strings.Join(wantStubs, ", ")) // ⛔ THE DISAGREEMENTS ARE REPORTED BEFORE THE ABSENCES, and that order is // the whole of it. Written the other way round -- the version-absence // refusal returning first -- this tool answered "found no statement of the @@ -513,9 +667,18 @@ func check(stdout, stderr io.Writer, root, baseline, runtimePath, shimPath strin for _, w := range wrongVersion { fmt.Fprintf(stderr, "%s: says the gate measures at %s, it measures at %s\n", w.where, w.got, wantVersion) } - if n := len(wrong) + len(wrongVersion); n > 0 { + for _, w := range wrongStub { + fmt.Fprintf(stderr, "%s: calls `%s` a stubbed shim helper; the shim stubs %s and not it, "+ + "so its examples RUN -- and when the behaviour is missing they FAIL rather than skip\n", + w.where, w.got, strings.Join(wantStubs, ", ")) + } + if n := len(wrong) + len(wrongVersion) + len(wrongStub); n > 0 { return &exitError{1, fmt.Sprintf("%d claim(s) disagree with the ruby repository", n)} } + // Now the held refusal, after every concrete finding has been printed. + if stubErr != nil { + return &exitError{2, fmt.Sprintf("cannot read the helpers the shim stubs: %v", stubErr)} + } if checked == 0 { return &exitError{1, fmt.Sprintf("found no statement of the total in %s; "+ "either the vocabulary no longer matches how the docs phrase it, or the figure was dropped", root)} @@ -528,6 +691,27 @@ func check(stdout, stderr io.Writer, root, baseline, runtimePath, shimPath strin return &exitError{1, fmt.Sprintf("found no statement of the version the gate measures at in %s; "+ "a total without it is a count over an unnamed population (Ruby %s today)", root, wantVersion)} } + // And the absence that matters most here: a page may name the stubbed + // helpers correctly and still be missing one, which is how `ruby_cmd` stayed + // unmentioned on both sites while `ruby_exe`, one line above it in the shim, + // was named everywhere. Naming a subset is the failure this cannot let pass. + // + // It applies only where the list is made at all: a tree that never names a + // stubbed helper has nothing to be incomplete about, and this tool is run + // over two trees of very different depth. That leaves one hole -- deleting + // the enumeration outright -- and it is closed by a test that runs this over + // the real content/en and requires the mentions to be there. + var unnamed []string + for _, s := range wantStubs { + if !namedSomewhere[s] { + unnamed = append(unnamed, s) + } + } + if stubsChecked > 0 && len(unnamed) > 0 { + return &exitError{1, fmt.Sprintf("the shim stubs %s, and %s names %s nowhere; "+ + "every page that explains why the total is a lower bound owes the reader the whole list", + strings.Join(wantStubs, ", "), root, strings.Join(unnamed, ", "))} + } return nil } @@ -675,6 +859,32 @@ func authoritativeVersion(runtimePath, shimPath string, timeout time.Duration) ( return runtime, "two copies, agreeing", nil } +// authoritativeStubs reports the mspec helpers the shim stubs, read out of +// spec_helper.rb itself. +// +// It fetches the shim a second time rather than sharing authoritativeVersion's +// copy on purpose: each of the two facts then refuses on its own terms, and a +// version check that cannot read the file does not also decide the stub list. +func authoritativeStubs(shimPath string, timeout time.Duration) ([]string, error) { + sh, err := fetch(shimPath, shimURL, timeout) + if err != nil { + return nil, fmt.Errorf("the shim's stubbed helpers: %w", err) + } + var out []string + for _, m := range shimStubRe.FindAllSubmatch(sh, -1) { + out = append(out, string(m[1])) + } + if len(out) == 0 { + // Not "nothing is stubbed": every page says the total is a lower bound + // BECAUSE something is, so an empty answer here means the pattern stopped + // matching the shim, and judging the pages against an empty set would + // turn every correct enumeration into a finding. + return nil, fmt.Errorf("no `def … raise SpecSkip` in %s; the pattern no longer matches the shim", shimURL) + } + sort.Strings(out) + return out, nil +} + // hasAnySuffix is spelled out so the file types this walks are a list in one // place rather than a chain of && that grows a bug each time it is extended. func hasAnySuffix(path string, suffixes ...string) bool { diff --git a/tools/conformancefigure/main_test.go b/tools/conformancefigure/main_test.go index 7ae7a94..90ff46d 100644 --- a/tools/conformancefigure/main_test.go +++ b/tools/conformancefigure/main_test.go @@ -3,8 +3,10 @@ package main import ( "bytes" "errors" + "io/fs" "os" "path/filepath" + "strings" "testing" "time" ) @@ -71,12 +73,30 @@ func versionFixtures(t *testing.T, runtimeV, shimV string) (string, string) { t.Fatal(err) } sp := filepath.Join(dir, "spec_helper.rb") - if err := os.WriteFile(sp, []byte("CUR_VERSION = \""+shimV+"\"\nPLATFORMS = [:darwin]\n"), 0o644); err != nil { + if err := os.WriteFile(sp, []byte("CUR_VERSION = \""+shimV+"\"\nPLATFORMS = [:darwin]\n"+shimStubs), 0o644); err != nil { t.Fatal(err) } return rp, sp } +// shimStubs is the part of spec_helper.rb the stubbed-helper check reads, in the +// exact shapes the real file uses: a bare `raise`, a `::Kernel.raise`, and -- +// the one that must NOT be read as a stub -- `skip`, which raises a SpecSkip +// SUBCLASS because the spec asked to be skipped. +// +// The implemented matchers are here too, with bodies, so a pattern that started +// matching any `def` in the file would be caught by a fixture rather than on +// the live site. +const shimStubs = ` +def output_to_fd(*a); raise SpecSkip, "output_to_fd matcher unsupported"; end +def be_computed_by(sym, *extra); BeComputedByMatcher.new(sym, *extra); end +def complain(pat = nil, options = {}); ComplainMatcher.new(pat, options[:verbose]); end +class SkippedSpecError < SpecSkip; end +def skip(reason = 'no reason'); ::Kernel.raise(SkippedSpecError, reason); end +def ruby_exe(*a, **k); ::Kernel.raise(SpecSkip, "ruby_exe subprocess unsupported"); end +def ruby_cmd(*a, **k); ::Kernel.raise(SpecSkip, "ruby_cmd unsupported"); end +` + // ⛔ An unreadable baseline is a refusal (2), never a pass: it finds no // disagreement either, and that silence reads exactly like agreement. func TestAnUnreadableBaselineRefuses(t *testing.T) { @@ -336,7 +356,7 @@ func TestADerivedShimVersionIsNotADisagreement(t *testing.T) { // With a trailing comment, because that is how the line will actually be // written: the change wants explaining at the site. if err := os.WriteFile(sp, - []byte("CUR_VERSION = RUBY_VERSION # one fact, one place (#799)\n"), 0o644); err != nil { + []byte("CUR_VERSION = RUBY_VERSION # one fact, one place (#799)\n"+shimStubs), 0o644); err != nil { t.Fatal(err) } @@ -461,3 +481,122 @@ func TestATomlFileIsWalked(t *testing.T) { t.Errorf("it said %q, want it to name the stale figure", said) } } + +// ⛔ The three cases the stubbed-helper check exists for, each one a state this +// site was actually in on 2026-11-09. +func TestTheStubbedHelperListIsJudged(t *testing.T) { + dir := t.TempDir() + base := filepath.Join(dir, "BASELINE") + if err := os.WriteFile(base, []byte("OK\ta_spec.rb\t23511\n"), 0o644); err != nil { + t.Fatal(err) + } + rp, sp := versionFixtures(t, "4.0.9", "4.0.9") + + // Every page carries a correct total and version, so nothing but the stub + // list can decide these cases. + const preamble = "It passes **23,511**, and those guards resolve at Ruby 4.0.9.\n" + + for _, tc := range []struct { + name string + page string + status int + says string + }{{ + name: "the list as this site carried it, naming two implemented matchers", + page: preamble + "The shim stubs a few MSpec matchers (`complain`, `output`, `ruby_exe`, …), so\n" + + "their examples count as skipped.\n", + status: 1, + says: "calls `complain` a stubbed shim helper", + }, { + name: "a list missing one, which is how `ruby_cmd` went unmentioned everywhere", + page: preamble + "The shim stubs two MSpec helpers — `ruby_exe` and `output_to_fd` — so their\n" + + "examples count as skipped.\n", + status: 1, + says: "names ruby_cmd nowhere", + }, { + name: "the whole list, and a sentence DENYING that the others are stubbed", + page: preamble + "The shim stubs `ruby_exe`, `ruby_cmd` and `output_to_fd`, so their examples\n" + + "skip. `complain`, `output` and `be_computed_by` are **implemented**, not\n" + + "stubbed; an example using a stubbed helper skips, one using `complain` runs.\n", + status: 0, + }} { + t.Run(tc.name, func(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "page.md"), []byte(tc.page), 0o644); err != nil { + t.Fatal(err) + } + var out, errb bytes.Buffer + cmd := RootCmd() + cmd.SetArgs([]string{"--root", root, "--baseline", base, + "--runtime", rp, "--shim", sp, "--timeout", "5s"}) + cmd.SetOut(&out) + cmd.SetErr(&errb) + err := cmd.Execute() + if tc.status == 0 { + if err != nil { + t.Fatalf("Execute: %v\nstdout: %s\nstderr: %s", err, out.String(), errb.String()) + } + return + } + var ee *exitError + if !errors.As(err, &ee) { + t.Fatalf("got %v, want an exitError", err) + } + if ee.code != tc.status { + t.Errorf("status %d, want %d", ee.code, tc.status) + } + said := ee.Error() + errb.String() + if !strings.Contains(said, tc.says) { + t.Errorf("it said %q, want it to name %q", said, tc.says) + } + }) + } +} + +// ⛔ THE HOLE THE WALK ITSELF CANNOT CLOSE. The completeness check only applies +// where the list is made at all, so deleting the enumeration outright would +// leave the tool with nothing to complete and a clean exit. This reads the real +// content/en and requires it to name every helper the shim stubs -- and only +// those -- independently of any figure on the page, so a figure refresh never +// makes this test noisy. +func TestTheSiteNamesEveryStubbedHelper(t *testing.T) { + want := map[string]bool{} + for _, m := range shimStubRe.FindAllStringSubmatch(shimStubs, -1) { + want[m[1]] = false + } + if len(want) != 3 { + t.Fatalf("the fixture shim yielded %d stubbed helper(s), want 3: the pattern and the fixture have drifted apart", len(want)) + } + + mentions := 0 + err := filepath.WalkDir("../../content/en", func(path string, d fs.DirEntry, err error) error { + if err != nil || d.IsDir() || !hasAnySuffix(path, ".md", ".html", ".toml") { + return err + } + body, rerr := os.ReadFile(path) + if rerr != nil { + return rerr + } + for i, line := range strings.Split(string(body), "\n") { + for _, name := range matchStubs(line) { + mentions++ + if _, ok := want[name]; !ok { + t.Errorf("%s:%d calls `%s` a stubbed shim helper; the shim stubs it nowhere", path, i+1, name) + continue + } + want[name] = true + } + } + return nil + }) + if err != nil { + t.Fatalf("walking content/en: %v", err) + } + for name, seen := range want { + if !seen { + t.Errorf("content/en names `%s` nowhere, and the shim stubs it: the page explains that the\n"+ + "total is a lower bound without saying which examples it is low by", name) + } + } + t.Logf("%d stubbed-helper mention(s) across content/en", mentions) +}