Repository navigation
fix(budget): make budget suspension stick and stop the team's RDS too - #183
Conversation
budget-enforcer now writes /javabin/budget-suspensions/{team}; the
ecs-service module plans desired_count=0 while it exists, so the next apply
no longer scales a suspended team back up. The enforcer also stops the
team's RDS instances, reconciles every 6 hours (re-stops RDS after its
7-day auto-start, re-zeros services) and lifts the suspension when spend is
back under 200% or on a manual {"action":"lift"} invoke, restoring the
previous desired counts and starting the stopped databases.
Also parse the plain-text AWS Budgets SNS message; the old handler only
accepted JSON and would have skipped every real notification.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Terraform Plan🚧 Changes detected — Plan: 3 to add, 2 to change, 0 to destroy. Plan outputLLM ReviewRisk: 🟡 MEDIUM Plan adds an EventBridge schedule (every 6 hours) that invokes the budget-enforcer Lambda, updates its IAM policy and environment variables, and redeploys its code; 0 destroys.
|
Combine the suspension marker with main's javabin:desired-count tag (#187): ecs-deploy now scales to the tag and Terraform ignores desired_count, so the ecs-service module plans the tag as "0" while the team is suspended. The enforcer pins the tag to 0 on suspend and reconcile, records the previous value in the marker, and restores it on lift. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merged
Checks: lambda tests 259 OK, scripts tests 184 OK, Behaviour change to look at: on main, raising the budget and running the app's pipeline brought the service back. With this branch, a tf-apply keeps the tag at |
Terraform Plan🚧 Changes detected — Plan: 6 to add, 11 to change, 0 to destroy. Plan outputLLM ReviewRisk: 🔴 HIGH Plan adds 6 resources (EventBridge rules, targets, Lambda permissions) and updates 11 in place (CI OIDC trust policies, ECS deploy and Lambda IAM policies, Lambda code and env vars); nothing is destroyed, but the CI trust policy changes need explicit sign-off.
|
Problem
budget-enforcerscaled a team's ECS services todesiredCount=0, butecs-servicesetsdesired_count = var.desired_count, so the team's next plan saw drift and the apply scaled the services back up. The team's RDS instances kept running and costing money. Nothing said how a suspension ends.While checking this I found a second problem: AWS Budgets sends plain-text SNS messages (
Budget Name: javabin-team-x…). The handler only accepted JSON, so a real 200% notification would have been logged as a parse error and skipped. The Lambda has never logged, so this has never run live.Change
/javabin/budget-suspensions/{team}(JSON: when it started, the budget, each service's previous desired count, the RDS instances it stopped). Then it scales the services to 0 and stops the team'savailableRDS instances. It skips Aurora members and replicas, because those can't be stopped one instance at a time.ecs-servicehonours it (budget.tf, plus a one-linedesired_countchange inmain.tf).aws_ssm_parameters_by_pathon the suspension path, anddesired_count = local.budget_suspended ? 0 : var.desired_count. I used this instead ofignore_changes = [desired_count], which would also stopapp.yamldesired_countchanges from ever applying. A by-path lookup returns an empty list when there is no marker, so plans don't fail.ci-plan(ReadOnlyAccess) andci-teamcan both read the marker: it is untagged, so the boundary'sDenyXTeamdoesn't apply.{"action":"reconcile"}). For each marker: if spend is still over 200% (or the budget can't be read), it scales any services that were scaled up by hand back to 0 and stops the databases again. RDS restarts a stopped instance by itself after 7 days, and this catches that. Otherwise (a new month, orbudget_nokraised in the registry) it lifts the suspension.{"action":"lift","team":"…"}invoke): deletes the marker, sets each recorded service that is still at 0 back to its previous count, starts the recorded RDS instances, and posts to Slack.rds:DescribeDBInstances,rds:Stop/StartDBInstanceon team-tagged instances only, andbudgets:ViewBudgetonjavabin-team-*.docs/cost-management.mdhas a new "Budget suspension" section on how a team gets unsuspended. Also updated:docs/lambda-functions.md,docs/reusable-modules.mdandCLAUDE.md.Known gap: the marker is not tamper-proof
The marker is untagged so every team's Terraform can read it. That also means a team CI role or SSO session could delete it. The right fix is a Deny in the developer boundary, but the live boundary is 6,098 of 6,144 characters, so there is no room. Tagging the marker
team=platformwould makeDenyXTeamblock teams from reading it too. Teams can already change their own budget and their own services' counts, so this is a safety net, not a security control. It is written down incost-management.md.Verification
python3 -m unittest discover -s terraform/lambda-src/tests: 228 OK. Newtest_budget_enforcer.pycovers: the plain-text and JSON message formats, suspend (marker, ECS, RDS, other teams untouched), reconcile re-applying after a manual scale-up and an RDS auto-start, lift on a raised budget (counts and RDS restored), keeping the suspension when the budget can't be read, manual lift, and lifting a team that isn't suspended (no-op).python3 -m unittest discover -s scripts/tests: 133 OK.terraform fmt -recursive terraform,terraform validateinterraform/modules/ecs-serviceandterraform/platform: valid.🤖 Generated with Claude Code