Skip to content

fix(status): stop a leftover ACME resolver from reporting divergence - #236

Merged
vishr merged 6 commits into
mainfrom
fix/status-stale-duplicate-cert
Nov 7, 2026
Merged

vishr merged 6 commits into
mainfrom
fix/status-stale-duplicate-cert

Conversation

@vishr

@vishr vishr commented Nov 7, 2026 •

Copy link
Copy Markdown
Member

What this changes

ob status reported divergence_detected and exited non-zero on a healthy host whose acme.json still held a certificate under a resolver key that an older proxy configuration left behind. Nothing renews that certificate, so once it entered the renewal floor, status diverged permanently (#235).

  • Status now reads the static configuration the host applied (traefik.yml or traefik.yaml in the host's proxy config directory) and maps each certificatesResolvers.*.acme.storage to its store file. It reads every store file a resolver names, plus acme.json and acme-wildcard.json.
  • A certificate can be overdue only when a resolver in that configuration writes its store. A certificate under any other key, or under a configured key in a store that resolver does not write, is listed as orphaned. It never sets Diverged. The text output names the resolver and the store's host path, but does not advise deleting the key: Traefik reads its static configuration only at startup, so a proxy not restarted after an apply may still renew under it.
  • A resolver whose store status cannot place (storage unset, outside /letsencrypt, or not a string) gets the benefit of the doubt: its certificates are judged by expiry and are never called orphaned. Keys and resolver names match without regard to case, a relative storage path resolves from /, and traefik.yaml is read before traefik.yml, as Traefik does.
  • If the applied configuration cannot be read or parsed, status records an issue (snapshot incomplete and diverged), judges every certificate by expiry alone, and calls none orphaned.
  • The JSON certificate entry gains resolver, store and orphaned. The overdue issue names the resolver and store. Both status views share one judging helper and one sort order.
  • /letsencrypt is one constant shared by the compose template, the default static configuration and the parser. The rendered bytes are unchanged, so no proxy restarts.

Dependency upgrades: xo/terminfo v1.3.1, go.yaml.in/yaml/v4 v4.0.0, astro 7.4.7, @astrojs/starlight 0.42.10, @astrojs/markdown-remark 7.3.6. spf13/pflag stays at v1.0.10, because v1.1.0 prints every boolean flag as --flag[=true|false] in help. TypeScript stays on 6.x.

Closes #235

Why this is correct

  • The applied configuration is what the running proxy renews by. Judging by the local configuration would call a resolver that was removed locally but still renews on the host a leftover, and the hint would tell the operator to delete a certificate the proxy still serves. TestStatusJudgesByAppliedConfigNotLocalConfig covers this.
  • Every uncertain case falls back to the pre-change behaviour (judge by expiry) rather than to "orphaned", so the change can remove false alarms but cannot hide a real overdue renewal or invite deleting a live certificate.
  • Tests in internal/engine/proxystatus_test.go, status_snapshot_test.go and internal/proxy/proxy_test.go cover: the store from the issue (text and JSON), a newer orphaned certificate that must not hide a failing managed one, a wildcard resolver removed from the configuration, a configured key in the wrong store, a custom store file name, an unparsable and an unreadable applied configuration, a resolver with an unplaceable store, and the parser's relative-path, case and escape handling.
  • Five review passes shaped this. Earlier approaches (newest certificate per domain, hardcoded resolver names, local configuration) each left a case that hid a real renewal failure, and those are the cases the tests now pin.

Known limits: a domain whose configured resolver never issued a certificate is still not reported, which needs routes compared with the store. Each status run makes up to two extra SSH round trips to read the applied configuration. Stale keys are reported, not pruned; having ob proxy apply prune resolver keys the applied configuration no longer defines would be the follow-up.

Effect on the safety envelope

None. Status reads one more host file and reports differently; nothing new is done to a running system.

Checklist

  • just check passes locally.
  • Tests cover the new behaviour, including the failure paths.
  • Generated documentation is current (just check verifies this).
  • I have accepted the CLA, or will when the bot asks on my first pull request.

vishr added 6 commits November 7, 2026 04:34
A resolver key left in acme.json by an older proxy configuration keeps a
certificate that nothing renews. ob status judged it like any other and
reported a healthy host as diverged once it entered the renewal floor.

Certificates now carry the resolver key and store file they came from.
Only certificates from letsencrypt and onebox-wildcard can be overdue.
Others are listed as orphaned, with the key to remove, and never set
Diverged. Both status views share one judging helper and one sort order.

Fixes #235
Go: xo/terminfo v1.3.1, go.yaml.in/yaml/v4 v4.0.0.
Site: astro 7.4.7, @astrojs/starlight 0.42.10, @astrojs/markdown-remark 7.3.6.

spf13/pflag stays at v1.0.10: v1.1.0 prints every boolean flag as
--flag[=true|false] in help output. TypeScript stays on 6.x.
…fines

Hardcoding letsencrypt and onebox-wildcard as managed left the #235
symptom in place for a host that dropped its wildcard routes, and for a
resolver key sitting in a store file it does not write.

Status now reads certificatesResolvers.*.acme.storage from the static
configuration it already stages, and renews-or-orphans each certificate
by its (resolver, store file) pair. Custom resolvers in a project-owned
traefik.yml are judged too. When the configuration cannot be read, no
certificate is judged overdue or orphaned. The orphan hint names the
store's full host path.
Judging by the locally staged config was wrong whenever the host ran an
older one: a resolver removed locally but still renewing on the host was
called orphaned, and the hint told the operator to delete a certificate
the proxy still serves.

Status now reads traefik.yml or traefik.yaml from the host's proxy config
directory in the certificate read and reads every store file a resolver
there names, as well as acme.json and acme-wildcard.json. When that file
is refused or does not parse, the issue is recorded, nothing is called
orphaned and certificates are judged by expiry alone, so a real overdue
renewal is never hidden. The local config read is back to its original
job. The /letsencrypt mount target is one constant shared by the compose
template, the static config and the parser.
A resolver with a relative storage path, mixed-case keys, or a storage
value status cannot place was treated as unconfigured, so its renewing
certificate was called orphaned with a hint to delete it.

CertificateStores now matches keys without regard to case, resolves a
relative storage path from /, and lists every defined resolver, mapping
an unplaceable store to "". Certificates of such a resolver are judged
by expiry and never orphaned. Any failed read of the applied config is
recorded, not raised, so the store reads in the same step still run.
The parse issue no longer quotes YAML errors, which can repeat values
from the file, and the overdue issue names the resolver and store.
Traefik reads its static configuration only at startup, so a proxy not
restarted after an apply can still renew under a key the file no longer
names. The orphaned hint now says the certificate is not renewed by the
applied configuration and names the resolver and store, without telling
the operator to delete it.

Resolver names match without regard to case. traefik.yaml is read before
traefik.yml, the order Traefik searches. The text view prints a config
issue before an unreadable store returns, as the JSON snapshot does, and
every config issue says certificates are judged by expiry alone.
@vishr
vishr merged commit 36fdd67 into main Nov 7, 2026
6 checks passed
@vishr
vishr deleted the fix/status-stale-duplicate-cert branch November 7, 2026 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ob status reports divergence for a stale certificate when a newer one exists

1 participant