Skip to content

Add alert rules to the chart and a runbook - #11

Merged
masih merged 3 commits into
mainfrom
masih/1794570563-alerts-runbook
Nov 13, 2026
Merged

masih merged 3 commits into
mainfrom
masih/1794570563-alerts-runbook

Conversation

@masih

@masih masih commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

The chart's ServiceMonitor scrapes the faucet's metrics, but nothing alerts on them, and there is no guide for an operator when something goes wrong.

prometheusRule.enabled adds a PrometheusRule with six alerts: the pod unreachable, /readyz failing (from kube-state-metrics), the balance below prometheusRule.lowBalanceSEI, a claim unsettled for over 10 minutes, repeated send errors or nonce resyncs, and failing RPC reads. Each alert links to its section of the new RUNBOOK.md, which also covers rotating the signer key and restoring the database. The rule needs the ServiceMonitor, so rendering fails if it is enabled alone. make verify now renders it, and promtool check rules accepts the rendered rules.

Two metric changes support the alerts. The new wellspring_oldest_unsettled_claim_seconds is the age of the oldest queued or sent claim, so the stuck alert also catches transfers the node accepted but never included, which wellspring_queue_depth misses. wellspring_balance_wei now has no series until the first balance read, so an RPC that is down from startup does not report an empty faucet.

@masih
masih marked this pull request as ready for review November 13, 2026 11:51
seidroid[bot]
seidroid Bot previously requested changes Nov 13, 2026

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds an opt-in PrometheusRule with six alerts and a RUNBOOK.md for the faucet. It's blocked by two problems: WellspringSenderStuck stays silent when the faucet is actually stuck, and the key-rotation procedure in the runbook leaves the sender in a bad state; the codex reading found nothing, so it added no findings.

Blocking

2 findings on the changed lines, as inline comments.

Non-blocking

2 findings on the changed lines, as inline comments.

1 nit, not posted on the code
  • RUNBOOK.md:52 — The sender already resyncs and rebroadcasts unsettled claims on its own once --confirm-timeout (30s) passes. If the alert is still firing after 10 minutes, a restart repeats what has already failed rather than fixing it.

seidroid review · decision request_changes · session 9ebbe49955714aa3826ed8e82771b543 · turn resp_claude_c0546da5ffb4263bddb0be89a6382aed · item 942d0cdffe735cc3ba04b45208959672

Findings: 2 blocking | 2 non-blocking | 4 posted inline

Comment thread chart/templates/prometheusrule.yaml Resolved
Comment thread RUNBOOK.md Resolved
Comment thread RUNBOOK.md Resolved
Comment thread chart/templates/prometheusrule.yaml Resolved

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the last review, a new wellspring_oldest_unsettled_claim_seconds gauge drives WellspringSenderStuck, the balance gauge has no series until the first read, and the runbook's key-rotation and restore steps now require no claim to be queued or sent. All four earlier findings are fixed and nothing new blocks; the codex reading found nothing, so it added no findings, and the tests were not run because the sandbox has no Go toolchain.

seidroid review · decision approve · session 9ebbe49955714aa3826ed8e82771b543 · turn resp_claude_984757d25c4b6f9d0c65d310f440ab2c · item 3fac87f3a08059ec9ab22055ecd97118

Findings: 0 blocking | 0 non-blocking | 0 posted inline

@seidroid
seidroid Bot dismissed their stale review November 13, 2026 12:00

Superseded: the latest review found nothing blocking in this change.

@masih
masih merged commit 45ee2a0 into main Nov 13, 2026
40 checks passed
@masih
masih deleted the masih/1794570563-alerts-runbook branch November 13, 2026 12:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant