Repository navigation
Add alert rules to the chart and a runbook - #11
Conversation
There was a problem hiding this comment.
This adds an opt-in PrometheusRule with six alerts and a RUNBOOK.md for the faucet. It's blocked by two problems: WellspringSenderStuck stays silent when the faucet is actually stuck, and the key-rotation procedure in the runbook leaves the sender in a bad state; the codex reading found nothing, so it added no findings.
Blocking
2 findings on the changed lines, as inline comments.
Non-blocking
2 findings on the changed lines, as inline comments.
1 nit, not posted on the code
RUNBOOK.md:52— The sender already resyncs and rebroadcasts unsettled claims on its own once--confirm-timeout(30s) passes. If the alert is still firing after 10 minutes, a restart repeats what has already failed rather than fixing it.
seidroid review · decision request_changes · session 9ebbe49955714aa3826ed8e82771b543 · turn resp_claude_c0546da5ffb4263bddb0be89a6382aed · item 942d0cdffe735cc3ba04b45208959672
Findings: 2 blocking | 2 non-blocking | 4 posted inline
There was a problem hiding this comment.
Since the last review, a new wellspring_oldest_unsettled_claim_seconds gauge drives WellspringSenderStuck, the balance gauge has no series until the first read, and the runbook's key-rotation and restore steps now require no claim to be queued or sent. All four earlier findings are fixed and nothing new blocks; the codex reading found nothing, so it added no findings, and the tests were not run because the sandbox has no Go toolchain.
seidroid review · decision approve · session 9ebbe49955714aa3826ed8e82771b543 · turn resp_claude_984757d25c4b6f9d0c65d310f440ab2c · item 3fac87f3a08059ec9ab22055ecd97118
Findings: 0 blocking | 0 non-blocking | 0 posted inline
Superseded: the latest review found nothing blocking in this change.
The chart's ServiceMonitor scrapes the faucet's metrics, but nothing alerts on them, and there is no guide for an operator when something goes wrong.
prometheusRule.enabledadds aPrometheusRulewith six alerts: the pod unreachable,/readyzfailing (from kube-state-metrics), the balance belowprometheusRule.lowBalanceSEI, a claim unsettled for over 10 minutes, repeated send errors or nonce resyncs, and failing RPC reads. Each alert links to its section of the newRUNBOOK.md, which also covers rotating the signer key and restoring the database. The rule needs the ServiceMonitor, so rendering fails if it is enabled alone.make verifynow renders it, andpromtool check rulesaccepts the rendered rules.Two metric changes support the alerts. The new
wellspring_oldest_unsettled_claim_secondsis the age of the oldest queued or sent claim, so the stuck alert also catches transfers the node accepted but never included, whichwellspring_queue_depthmisses.wellspring_balance_weinow has no series until the first balance read, so an RPC that is down from startup does not report an empty faucet.