Skip to content

fix(budget): make budget suspension stick and stop the team's RDS too - #183

Merged
Alexanderamiri merged 2 commits into
mainfrom
fix/budget-suspend-sticks
Oct 31, 2026
Merged

Alexanderamiri merged 2 commits into
mainfrom
fix/budget-suspend-sticks

Conversation

@Alexanderamiri

Copy link
Copy Markdown
Member

Problem

budget-enforcer scaled a team's ECS services to desiredCount=0, but ecs-service sets desired_count = var.desired_count, so the team's next plan saw drift and the apply scaled the services back up. The team's RDS instances kept running and costing money. Nothing said how a suspension ends.

While checking this I found a second problem: AWS Budgets sends plain-text SNS messages (Budget Name: javabin-team-x …). The handler only accepted JSON, so a real 200% notification would have been logged as a parse error and skipped. The Lambda has never logged, so this has never run live.

Change

  • Suspension marker. On a 200% notification the enforcer writes /javabin/budget-suspensions/{team} (JSON: when it started, the budget, each service's previous desired count, the RDS instances it stopped). Then it scales the services to 0 and stops the team's available RDS instances. It skips Aurora members and replicas, because those can't be stopped one instance at a time.
  • ecs-service honours it (budget.tf, plus a one-line desired_count change in main.tf). aws_ssm_parameters_by_path on the suspension path, and desired_count = local.budget_suspended ? 0 : var.desired_count. I used this instead of ignore_changes = [desired_count], which would also stop app.yaml desired_count changes from ever applying. A by-path lookup returns an empty list when there is no marker, so plans don't fail. ci-plan (ReadOnlyAccess) and ci-team can both read the marker: it is untagged, so the boundary's DenyXTeam doesn't apply.
  • Reconcile every 6 hours (EventBridge, {"action":"reconcile"}). For each marker: if spend is still over 200% (or the budget can't be read), it scales any services that were scaled up by hand back to 0 and stops the databases again. RDS restarts a stopped instance by itself after 7 days, and this catches that. Otherwise (a new month, or budget_nok raised in the registry) it lifts the suspension.
  • Lift (from reconcile or a manual {"action":"lift","team":"…"} invoke): deletes the marker, sets each recorded service that is still at 0 back to its previous count, starts the recorded RDS instances, and posts to Slack.
  • IAM for the enforcer: SSM on the marker path, rds:DescribeDBInstances, rds:Stop/StartDBInstance on team-tagged instances only, and budgets:ViewBudget on javabin-team-*.
  • Docs: docs/cost-management.md has a new "Budget suspension" section on how a team gets unsuspended. Also updated: docs/lambda-functions.md, docs/reusable-modules.md and CLAUDE.md.

Known gap: the marker is not tamper-proof

The marker is untagged so every team's Terraform can read it. That also means a team CI role or SSO session could delete it. The right fix is a Deny in the developer boundary, but the live boundary is 6,098 of 6,144 characters, so there is no room. Tagging the marker team=platform would make DenyXTeam block teams from reading it too. Teams can already change their own budget and their own services' counts, so this is a safety net, not a security control. It is written down in cost-management.md.

Verification

  • python3 -m unittest discover -s terraform/lambda-src/tests: 228 OK. New test_budget_enforcer.py covers: the plain-text and JSON message formats, suspend (marker, ECS, RDS, other teams untouched), reconcile re-applying after a manual scale-up and an RDS auto-start, lift on a raised budget (counts and RDS restored), keeping the suspension when the budget can't be read, manual lift, and lifting a team that isn't suspended (no-op).
  • python3 -m unittest discover -s scripts/tests: 133 OK.
  • terraform fmt -recursive terraform, terraform validate in terraform/modules/ecs-service and terraform/platform: valid.
  • Live (read-only): the team budgets have the 200% ACTUAL notification, there are no markers yet, and the developer boundary is 6,098 characters.

🤖 Generated with Claude Code

budget-enforcer now writes /javabin/budget-suspensions/{team}; the
ecs-service module plans desired_count=0 while it exists, so the next apply
no longer scales a suspended team back up. The enforcer also stops the
team's RDS instances, reconciles every 6 hours (re-stops RDS after its
7-day auto-start, re-zeros services) and lifts the suspension when spend is
back under 200% or on a manual {"action":"lift"} invoke, restoring the
previous desired counts and starting the stopped databases.

Also parse the plain-text AWS Budgets SNS message; the old handler only
accepted JSON and would have skipped every real notification.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Alexanderamiri
Alexanderamiri requested a review from a team as a code owner October 31, 2026 12:07
@github-actions

Copy link
Copy Markdown

Terraform Plan

🚧 Changes detected — Plan: 3 to add, 2 to change, 0 to destroy.

Plan output
Acquiring state lock. This may take a few moments...
aws_ssm_parameter.atlassian_org_id: Refreshing state... [id=/javabin/platform/atlassian-org-id]
aws_ssm_parameter.atlassian_api_key: Refreshing state... [id=/javabin/platform/atlassian-api-key]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # module.lambdas.aws_cloudwatch_event_rule.budget_enforcer_reconcile will be created
  + resource "aws_cloudwatch_event_rule" "budget_enforcer_reconcile" {
      + arn                 = (known after apply)
      + description         = "Reconcile team budget suspensions every 6 hours"
      + event_bus_name      = "default"
      + force_destroy       = false
      + id                  = (known after apply)
      + name                = "javabin-budget-enforcer-reconcile"
      + name_prefix         = (known after apply)
      + schedule_expression = "rate(6 hours)"
      + tags_all            = {
          + "environment" = "production"
          + "managed-by"  = "terraform"
          + "repo"        = "javaBin/platform"
          + "service"     = "platform"
          + "team"        = "platform"
        }
    }

  # module.lambdas.aws_cloudwatch_event_target.budget_enforcer_reconcile will be created
  + resource "aws_cloudwatch_event_target" "budget_enforcer_reconcile" {
      + arn            = "arn:aws:lambda:eu-central-1:553637109631:function:javabin-budget-enforcer"
      + event_bus_name = "default"
      + force_destroy  = false
      + id             = (known after apply)
      + input          = jsonencode(
            {
              + action = "reconcile"
            }
        )
      + rule           = "javabin-budget-enforcer-reconcile"
      + target_id      = "reconcile-budget-enforcer"
    }

  # module.lambdas.aws_iam_role_policy.budget_enforcer will be updated in-place
  ~ resource "aws_iam_role_policy" "budget_enforcer" {
        id     = "javabin-budget-enforcer:javabin-budget-enforcer"
        name   = "javabin-budget-enforcer"
      ~ policy = jsonencode(
          ~ {
              ~ Statement = [
                    # (3 unchanged elements hidden)
                    {
                        Action   = "ecs:UpdateService"
                        Effect   = "Allow"
                        Resource = "arn:aws:ecs:eu-central-1:553637109631:service/javabin-platform/*"
                        Sid      = "ECSUpdateService"
                    },
                  + {
                      + Action   = [
                          + "ssm:GetParameter",
                          + "ssm:GetParametersByPath",
                          + "ssm:PutParameter",
                          + "ssm:DeleteParameter",
                        ]
                      + Effect   = "Allow"
                      + Resource = [
                          + "arn:aws:ssm:eu-central-1:553637109631:parameter/javabin/budget-suspensions",
                          + "arn:aws:ssm:eu-central-1:553637109631:parameter/javabin/budget-suspensions/*",
                        ]
                      + Sid      = "SuspensionMarkers"
                    },
                  + {
                      + Action   = "rds:DescribeDBInstances"
                      + Effect   = "Allow"
                      + Resource = "arn:aws:rds:eu-central-1:553637109631:db:*"
                      + Sid      = "RDSDescribe"
                    },
                  + {
                      + Action    = [
                          + "rds:StopDBInstance",
                          + "rds:StartDBInstance",
                        ]
                      + Condition = {
                          + Null = {
                              + "aws:ResourceTag/team" = "false"
                            }
                        }
                      + Effect    = "Allow"
                      + Resource  = "arn:aws:rds:eu-central-1:553637109631:db:*"
                      + Sid       = "RDSStopStart"
                    },
                  + {
                      + Action   = "budgets:ViewBudget"
                      + Effect   = "Allow"
                      + Resource = "arn:aws:budgets::553637109631:budget/javabin-team-*"
                      + Sid      = "BudgetsRead"
                    },
                ]
                # (1 unchanged attribute hidden)
            }
        )
        # (1 unchanged attribute hidden)
    }

  # module.lambdas.aws_lambda_function.budget_enforcer will be updated in-place
  ~ resource "aws_lambda_function" "budget_enforcer" {
        id                             = "javabin-budget-enforcer"
      ~ last_modified                  = "2026-03-26T20:01:54.976+0000" -> (known after apply)
      ~ source_code_hash               = "i3E4tI5mT0qCS601izSv52QOgwAndKfcZ3JkLzxTNis=" -> "Wn8w9m+w5RMlPZEVL6oilXPl3QiNairhkLb1bkuiyAo="
        tags                           = {}
        # (21 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              + "AWS_ACCOUNT_ID"     = "553637109631"
              + "SUSPENSION_PATH"    = "/javabin/budget-suspensions"
                # (3 unchanged elements hidden)
            }
        }

        # (3 unchanged blocks hidden)
    }

  # module.lambdas.aws_lambda_permission.budget_enforcer_reconcile will be created
  + resource "aws_lambda_permission" "budget_enforcer_reconcile" {
      + action              = "lambda:InvokeFunction"
      + function_name       = "javabin-budget-enforcer"
      + id                  = (known after apply)
      + principal           = "events.amazonaws.com"
      + source_arn          = (known after apply)
      + statement_id        = "AllowEventBridgeReconcile"
      + statement_id_prefix = (known after apply)
    }

Plan: 3 to add, 2 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

LLM Review

Risk: 🟡 MEDIUM

Plan adds an EventBridge schedule (every 6 hours) that invokes the budget-enforcer Lambda, updates its IAM policy and environment variables, and redeploys its code; 0 destroys.

  • 🔒 [security] The budget_enforcer role policy gains new permissions: ssm Get/GetParametersByPath/Put/Delete scoped to /javabin/budget-suspensions paths, and rds:StopDBInstance/StartDBInstance on all DB instances in the account. The RDS stop/start grant is conditioned on aws:ResourceTag/team existing (Null = false), so it is not limited to any specific team tag value. This is a broad power to stop production databases; scoping it to specific tag values or specific DB ARNs would reduce blast radius.
  • 🔒 [security] The new RDS stop/start permission applies to any DB instance that has a team tag, including production databases. A bug or misuse in the reconcile logic could stop production databases. Confirm the Lambda's logic only targets intended instances and that a stopped RDS instance will be restarted by AWS (RDS auto-starts stopped instances after 7 days).
  • 💰 [cost] The budget enforcer's purpose is to suspend spending, so it is intended to reduce cost. The EventBridge rule, target and permission are cheap. No NAT gateways, load balancers or new RDS/ECS resources are created.
  • ✅ [routine] The Lambda code hash changes and the environment gains AWS_ACCOUNT_ID and SUSPENSION_PATH variables. The rule, target and invoke permission are new (scheduled reconcile every 6 hours). The permission is scoped to events.amazonaws.com without a source_arn restriction in the plan (source_arn is known after apply, so it will be bound to the rule).
  • 💥 [destruction] No resources are destroyed (0 to destroy). The IAM policy update is in-place and only adds statements; existing ECS UpdateService statement is unchanged.

Combine the suspension marker with main's javabin:desired-count tag (#187):
ecs-deploy now scales to the tag and Terraform ignores desired_count, so the
ecs-service module plans the tag as "0" while the team is suspended. The
enforcer pins the tag to 0 on suspend and reconcile, records the previous
value in the marker, and restores it on lift.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Alexanderamiri

Copy link
Copy Markdown
Member Author

Merged origin/main (#184, #178, #180, #187, #186, #185) in 1eb7928. How each conflict was resolved:

  • terraform/modules/ecs-service/main.tf: kept main's desired_count = 0 and ignore_changes = [task_definition, desired_count] (fix(deploy): run the commit's image digest and fail on ECS rollback #187). Since fix(deploy): run the commit's image digest and fail on ECS rollback #187, ecs-deploy scales each service to its javabin:desired-count tag, so this PR's local.budget_suspended ? 0 : var.desired_count would no longer have any effect. The marker now decides the tag instead: "javabin:desired-count" = local.budget_suspended ? "0" : tostring(var.desired_count). I updated the comment in budget.tf to match.
  • budget_enforcer/handler.py: kept this PR's structure (marker, reconcile, lift, RDS, plain-text parsing). From main's version I kept the part that pins the tag to "0", and moved it into scale_team_to_zero. That includes services with 0 tasks whose tag isn't 0 yet (not deployed yet). New: the previous tag value goes into the marker as desired_count_tags, reconcile re-pins a tag someone reset by hand, and lift puts the tag back if it is still "0". Without that, the deploy after a lift would scale the service straight back to 0. Main's Slack text ("run the app's pipeline to restore") is replaced by this PR's lift instructions, which now mention the tag.
  • tests/test_budget_enforcer.py (both sides added it): kept this PR's fakes and ported main's three tag cases into them: pin on suspend, pin on a service not deployed yet, leave services that are already pinned or untagged alone. Added two more cases: lift restores the tag, and reconcile re-pins a tag reset by hand.
  • CLAUDE.md, docs/cost-management.md, docs/reusable-modules.md: kept both sides. The Lambda table rows, the "every 3 hours" team-provisioner flow lines from main, and main's Containers and grace-period/exec paragraphs all stay. The budget text now describes the tag instead of desired_count.
  • I also changed some text that didn't conflict but described main's "until a tf-apply writes it back" behaviour: docs/lambda-functions.md, docs/reusable-workflows.md, and the header comment and 0-count warning in scripts/ecs-deploy.sh. The test still finds "budget-enforcer" in that warning.

Checks: lambda tests 259 OK, scripts tests 184 OK, terraform fmt -check -recursive clean, terraform validate passes in terraform/platform and terraform/modules/ecs-service.

Behaviour change to look at: on main, raising the budget and running the app's pipeline brought the service back. With this branch, a tf-apply keeps the tag at "0" until the marker is gone. Reconcile (within 6 h) or a manual lift is what restores it.

@github-actions

Copy link
Copy Markdown

Terraform Plan

🚧 Changes detected — Plan: 6 to add, 11 to change, 0 to destroy.

Plan output
Acquiring state lock. This may take a few moments...
aws_ssm_parameter.atlassian_api_key: Refreshing state... [id=/javabin/platform/atlassian-api-key]
aws_ssm_parameter.atlassian_org_id: Refreshing state... [id=/javabin/platform/atlassian-org-id]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # module.iam.aws_iam_role.ci_infra will be updated in-place
  ~ resource "aws_iam_role" "ci_infra" {
      ~ assume_role_policy    = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Condition = {
                          ~ StringLike   = {
                              ~ "token.actions.githubusercontent.com:sub"              = [
                                    # (1 unchanged element hidden)
                                    "repo:javaBin/platform:environment:production",
                                  + "repo:javaBin@95047/platform@1175354559:ref:refs/heads/main",
                                  + "repo:javaBin@95047/platform@1175354559:environment:production",
                                ]
                                # (1 unchanged attribute hidden)
                            }
                            # (1 unchanged attribute hidden)
                        }
                        # (3 unchanged attributes hidden)
                    },
                ]
                # (1 unchanged attribute hidden)
            }
        )
        id                    = "javabin-ci-infra"
        name                  = "javabin-ci-infra"
        tags                  = {
            "Name" = "javabin-ci-infra"
        }
        # (9 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

  # module.iam.aws_iam_role.ci_infra_plan will be updated in-place
  ~ resource "aws_iam_role" "ci_infra_plan" {
      ~ assume_role_policy    = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Condition = {
                          ~ StringLike   = {
                              ~ "token.actions.githubusercontent.com:sub" = "repo:javaBin/platform:*" -> [
                                  + "repo:javaBin/platform:*",
                                  + "repo:javaBin@95047/platform@1175354559:*",
                                ]
                            }
                            # (1 unchanged attribute hidden)
                        }
                        # (3 unchanged attributes hidden)
                    },
                ]
                # (1 unchanged attribute hidden)
            }
        )
        id                    = "javabin-ci-infra-plan"
        name                  = "javabin-ci-infra-plan"
        tags                  = {
            "Name" = "javabin-ci-infra-plan"
        }
        # (9 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # module.iam.aws_iam_role.ci_registry will be updated in-place
  ~ resource "aws_iam_role" "ci_registry" {
      ~ assume_role_policy    = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Condition = {
                          ~ StringLike   = {
                              ~ "token.actions.githubusercontent.com:sub" = "repo:javaBin/registry:ref:refs/heads/main" -> [
                                  + "repo:javaBin/registry:ref:refs/heads/main",
                                  + "repo:javaBin@95047/registry@1175355770:ref:refs/heads/main",
                                ]
                            }
                            # (1 unchanged attribute hidden)
                        }
                        # (3 unchanged attributes hidden)
                    },
                ]
                # (1 unchanged attribute hidden)
            }
        )
        id                    = "javabin-ci-registry"
        name                  = "javabin-ci-registry"
        tags                  = {
            "Name" = "javabin-ci-registry"
        }
        # (9 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # module.iam.aws_iam_role_policy.ci_deploy_ecs["cupcake"] will be updated in-place
  ~ resource "aws_iam_role_policy" "ci_deploy_ecs" {
        id     = "javabin-ci-deploy-cupcake:ecs-deploy"
        name   = "ecs-deploy"
      ~ policy = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Action    = [
                            # (1 unchanged element hidden)
                            "ecs:DescribeServices",
                          + "ecs:ListTagsForResource",
                          + "ecs:DescribeTasks",
                        ]
                        # (4 unchanged attributes hidden)
                    },
                  + {
                      + Action    = [
                          + "ecs:ListTasks",
                        ]
                      + Condition = {
                          + ArnEquals = {
                              + "ecs:cluster" = "arn:aws:ecs:eu-central-1:553637109631:cluster/javabin-platform"
                            }
                        }
                      + Effect    = "Allow"
                      + Resource  = "*"
                      + Sid       = "EcsListTasks"
                    },
                  + {
                      + Action    = [
                          + "ecs:TagResource",
                        ]
                      + Condition = {
                          + StringEquals = {
                              + "ecs:CreateAction" = "RegisterTaskDefinition"
                            }
                        }
                      + Effect    = "Allow"
                      + Resource  = "*"
                      + Sid       = "EcsTagOnRegister"
                    },
                    {
                        Action   = [
                            "ecs:DescribeTaskDefinition",
                            "ecs:RegisterTaskDefinition",
                            "ecs:DeregisterTaskDefinition",
                        ]
                        Effect   = "Allow"
                        Resource = "*"
                        Sid      = "EcsTaskDefinitionManagement"
                    },
                    # (1 unchanged element hidden)
                ]
                # (1 unchanged attribute hidden)
            }
        )
        # (1 unchanged attribute hidden)
    }

  # module.iam.aws_iam_role_policy.ci_deploy_ecs["javazoneweb"] will be updated in-place
  ~ resource "aws_iam_role_policy" "ci_deploy_ecs" {
        id     = "javabin-ci-deploy-javazoneweb:ecs-deploy"
        name   = "ecs-deploy"
      ~ policy = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Action    = [
                            # (1 unchanged element hidden)
                            "ecs:DescribeServices",
                          + "ecs:ListTagsForResource",
                          + "ecs:DescribeTasks",
                        ]
                        # (4 unchanged attributes hidden)
                    },
                  + {
                      + Action    = [
                          + "ecs:ListTasks",
                        ]
                      + Condition = {
                          + ArnEquals = {
                              + "ecs:cluster" = "arn:aws:ecs:eu-central-1:553637109631:cluster/javabin-platform"
                            }
                        }
                      + Effect    = "Allow"
                      + Resource  = "*"
                      + Sid       = "EcsListTasks"
                    },
                  + {
                      + Action    = [
                          + "ecs:TagResource",
                        ]
                      + Condition = {
                          + StringEquals = {
                              + "ecs:CreateAction" = "RegisterTaskDefinition"
                            }
                        }
                      + Effect    = "Allow"
                      + Resource  = "*"
                      + Sid       = "EcsTagOnRegister"
                    },
                    {
                        Action   = [
                            "ecs:DescribeTaskDefinition",
                            "ecs:RegisterTaskDefinition",
                            "ecs:DeregisterTaskDefinition",
                        ]
                        Effect   = "Allow"
                        Resource = "*"
                        Sid      = "EcsTaskDefinitionManagement"
                    },
                    # (1 unchanged element hidden)
                ]
                # (1 unchanged attribute hidden)
            }
        )
        # (1 unchanged attribute hidden)
    }

  # module.iam.aws_iam_role_policy.ci_deploy_ecs["jzwatch"] will be updated in-place
  ~ resource "aws_iam_role_policy" "ci_deploy_ecs" {
        id     = "javabin-ci-deploy-jzwatch:ecs-deploy"
        name   = "ecs-deploy"
      ~ policy = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Action    = [
                            # (1 unchanged element hidden)
                            "ecs:DescribeServices",
                          + "ecs:ListTagsForResource",
                          + "ecs:DescribeTasks",
                        ]
                        # (4 unchanged attributes hidden)
                    },
                  + {
                      + Action    = [
                          + "ecs:ListTasks",
                        ]
                      + Condition = {
                          + ArnEquals = {
                              + "ecs:cluster" = "arn:aws:ecs:eu-central-1:553637109631:cluster/javabin-platform"
                            }
                        }
                      + Effect    = "Allow"
                      + Resource  = "*"
                      + Sid       = "EcsListTasks"
                    },
                  + {
                      + Action    = [
                          + "ecs:TagResource",
                        ]
                      + Condition = {
                          + StringEquals = {
                              + "ecs:CreateAction" = "RegisterTaskDefinition"
                            }
                        }
                      + Effect    = "Allow"
                      + Resource  = "*"
                      + Sid       = "EcsTagOnRegister"
                    },
                    {
                        Action   = [
                            "ecs:DescribeTaskDefinition",
                            "ecs:RegisterTaskDefinition",
                            "ecs:DeregisterTaskDefinition",
                        ]
                        Effect   = "Allow"
                        Resource = "*"
                        Sid      = "EcsTaskDefinitionManagement"
                    },
                    # (1 unchanged element hidden)
                ]
                # (1 unchanged attribute hidden)
            }
        )
        # (1 unchanged attribute hidden)
    }

  # module.iam.aws_iam_role_policy.ci_deploy_ecs["testteam"] will be updated in-place
  ~ resource "aws_iam_role_policy" "ci_deploy_ecs" {
        id     = "javabin-ci-deploy-testteam:ecs-deploy"
        name   = "ecs-deploy"
      ~ policy = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Action    = [
                            # (1 unchanged element hidden)
                            "ecs:DescribeServices",
                          + "ecs:ListTagsForResource",
                          + "ecs:DescribeTasks",
                        ]
                        # (4 unchanged attributes hidden)
                    },
                  + {
                      + Action    = [
                          + "ecs:ListTasks",
                        ]
                      + Condition = {
                          + ArnEquals = {
                              + "ecs:cluster" = "arn:aws:ecs:eu-central-1:553637109631:cluster/javabin-platform"
                            }
                        }
                      + Effect    = "Allow"
                      + Resource  = "*"
                      + Sid       = "EcsListTasks"
                    },
                  + {
                      + Action    = [
                          + "ecs:TagResource",
                        ]
                      + Condition = {
                          + StringEquals = {
                              + "ecs:CreateAction" = "RegisterTaskDefinition"
                            }
                        }
                      + Effect    = "Allow"
                      + Resource  = "*"
                      + Sid       = "EcsTagOnRegister"
                    },
                    {
                        Action   = [
                            "ecs:DescribeTaskDefinition",
                            "ecs:RegisterTaskDefinition",
                            "ecs:DeregisterTaskDefinition",
                        ]
                        Effect   = "Allow"
                        Resource = "*"
                        Sid      = "EcsTaskDefinitionManagement"
                    },
                    # (1 unchanged element hidden)
                ]
                # (1 unchanged attribute hidden)
            }
        )
        # (1 unchanged attribute hidden)
    }

  # module.lambdas.aws_cloudwatch_event_rule.budget_enforcer_reconcile will be created
  + resource "aws_cloudwatch_event_rule" "budget_enforcer_reconcile" {
      + arn                 = (known after apply)
      + description         = "Reconcile team budget suspensions every 6 hours"
      + event_bus_name      = "default"
      + force_destroy       = false
      + id                  = (known after apply)
      + name                = "javabin-budget-enforcer-reconcile"
      + name_prefix         = (known after apply)
      + schedule_expression = "rate(6 hours)"
      + tags_all            = {
          + "environment" = "production"
          + "managed-by"  = "terraform"
          + "repo"        = "javaBin/platform"
          + "service"     = "platform"
          + "team"        = "platform"
        }
    }

  # module.lambdas.aws_cloudwatch_event_rule.team_provisioner_access_schedule will be created
  + resource "aws_cloudwatch_event_rule" "team_provisioner_access_schedule" {
      + arn                 = (known after apply)
      + description         = "Retry Identity Center group memberships for members Google SCIM has since provisioned"
      + event_bus_name      = "default"
      + force_destroy       = false
      + id                  = (known after apply)
      + name                = "javabin-team-provisioner-access-schedule"
      + name_prefix         = (known after apply)
      + schedule_expression = "rate(3 hours)"
      + tags_all            = {
          + "environment" = "production"
          + "managed-by"  = "terraform"
          + "repo"        = "javaBin/platform"
          + "service"     = "platform"
          + "team"        = "platform"
        }
    }

  # module.lambdas.aws_cloudwatch_event_target.budget_enforcer_reconcile will be created
  + resource "aws_cloudwatch_event_target" "budget_enforcer_reconcile" {
      + arn            = "arn:aws:lambda:eu-central-1:553637109631:function:javabin-budget-enforcer"
      + event_bus_name = "default"
      + force_destroy  = false
      + id             = (known after apply)
      + input          = jsonencode(
            {
              + action = "reconcile"
            }
        )
      + rule           = "javabin-budget-enforcer-reconcile"
      + target_id      = "reconcile-budget-enforcer"
    }

  # module.lambdas.aws_cloudwatch_event_target.team_provisioner_access will be created
  + resource "aws_cloudwatch_event_target" "team_provisioner_access" {
      + arn            = "arn:aws:lambda:eu-central-1:553637109631:function:javabin-team-provisioner"
      + event_bus_name = "default"
      + force_destroy  = false
      + id             = (known after apply)
      + input          = jsonencode(
            {
              + action = "sync_access_members"
            }
        )
      + rule           = "javabin-team-provisioner-access-schedule"
      + target_id      = "invoke-team-provisioner-access"
    }

  # module.lambdas.aws_iam_role_policy.budget_enforcer will be updated in-place
  ~ resource "aws_iam_role_policy" "budget_enforcer" {
        id     = "javabin-budget-enforcer:javabin-budget-enforcer"
        name   = "javabin-budget-enforcer"
      ~ policy = jsonencode(
          ~ {
              ~ Statement = [
                    # (2 unchanged elements hidden)
                    {
                        Action   = "ecs:ListServices"
                        Effect   = "Allow"
                        Resource = "arn:aws:ecs:eu-central-1:553637109631:cluster/javabin-platform"
                        Sid      = "ECSListServicesUnconstrained"
                    },
                  ~ {
                      ~ Action   = "ecs:UpdateService" -> [
                          + "ecs:UpdateService",
                          + "ecs:TagResource",
                        ]
                        # (3 unchanged attributes hidden)
                    },
                  + {
                      + Action   = [
                          + "ssm:GetParameter",
                          + "ssm:GetParametersByPath",
                          + "ssm:PutParameter",
                          + "ssm:DeleteParameter",
                        ]
                      + Effect   = "Allow"
                      + Resource = [
                          + "arn:aws:ssm:eu-central-1:553637109631:parameter/javabin/budget-suspensions",
                          + "arn:aws:ssm:eu-central-1:553637109631:parameter/javabin/budget-suspensions/*",
                        ]
                      + Sid      = "SuspensionMarkers"
                    },
                  + {
                      + Action   = "rds:DescribeDBInstances"
                      + Effect   = "Allow"
                      + Resource = "arn:aws:rds:eu-central-1:553637109631:db:*"
                      + Sid      = "RDSDescribe"
                    },
                  + {
                      + Action    = [
                          + "rds:StopDBInstance",
                          + "rds:StartDBInstance",
                        ]
                      + Condition = {
                          + Null = {
                              + "aws:ResourceTag/team" = "false"
                            }
                        }
                      + Effect    = "Allow"
                      + Resource  = "arn:aws:rds:eu-central-1:553637109631:db:*"
                      + Sid       = "RDSStopStart"
                    },
                  + {
                      + Action   = "budgets:ViewBudget"
                      + Effect   = "Allow"
                      + Resource = "arn:aws:budgets::553637109631:budget/javabin-team-*"
                      + Sid      = "BudgetsRead"
                    },
                ]
                # (1 unchanged attribute hidden)
            }
        )
        # (1 unchanged attribute hidden)
    }

  # module.lambdas.aws_iam_role_policy.team_provisioner will be updated in-place
  ~ resource "aws_iam_role_policy" "team_provisioner" {
        id     = "javabin-team-provisioner:javabin-team-provisioner"
        name   = "javabin-team-provisioner"
      ~ policy = jsonencode(
          ~ {
              ~ Statement = [
                    # (4 unchanged elements hidden)
                    {
                        Action   = [
                            "identitystore:CreateGroup",
                            "identitystore:DescribeGroup",
                            "identitystore:DescribeUser",
                            "identitystore:ListGroups",
                            "identitystore:CreateGroupMembership",
                            "identitystore:ListGroupMemberships",
                            "identitystore:DeleteGroupMembership",
                            "identitystore:ListUsers",
                        ]
                        Effect   = "Allow"
                        Resource = "*"
                        Sid      = "IdentityStoreSync"
                    },
                  ~ {
                      ~ Action   = [
                            # (1 unchanged element hidden)
                            "sso:DescribeAccountAssignmentCreationStatus",
                          + "sso:ListAccountAssignments",
                            "sso:ListPermissionSets",
                            # (7 unchanged elements hidden)
                        ]
                        # (3 unchanged attributes hidden)
                    },
                    {
                        Action   = [
                            "iam:GetPolicy",
                            "iam:GetPolicyVersion",
                            "iam:ListPolicyTags",
                        ]
                        Effect   = "Allow"
                        Resource = "arn:aws:iam::553637109631:policy/javabin-team-*-guardrails"
                        Sid      = "ReadTeamGuardrails"
                    },
                    # (1 unchanged element hidden)
                ]
                # (1 unchanged attribute hidden)
            }
        )
        # (1 unchanged attribute hidden)
    }

  # module.lambdas.aws_lambda_function.budget_enforcer will be updated in-place
  ~ resource "aws_lambda_function" "budget_enforcer" {
        id                             = "javabin-budget-enforcer"
      ~ last_modified                  = "2026-03-26T20:01:54.976+0000" -> (known after apply)
      ~ source_code_hash               = "i3E4tI5mT0qCS601izSv52QOgwAndKfcZ3JkLzxTNis=" -> "di7/qFyHX8tYBT6OXRTbwov0mZQqBuJuNY4jdUtzM2E="
        tags                           = {}
        # (21 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              + "AWS_ACCOUNT_ID"     = "553637109631"
              + "SUSPENSION_PATH"    = "/javabin/budget-suspensions"
                # (3 unchanged elements hidden)
            }
        }

        # (3 unchanged blocks hidden)
    }

  # module.lambdas.aws_lambda_function.team_provisioner will be updated in-place
  ~ resource "aws_lambda_function" "team_provisioner" {
        id                             = "javabin-team-provisioner"
      ~ last_modified                  = "2026-10-31T13:03:32.000+0000" -> (known after apply)
      ~ source_code_hash               = "3I4td1T2ENiRLUUEjeBLPinRaiXc/sZrwOT+Ai83lOY=" -> "4Gcog5O29Hs6CEnMQPK5WWLQ1aShJnqsS3zVkqRjVlA="
        tags                           = {}
        # (21 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              + "AWS_ACCESS_GROUP"             = "javabin-developer@java.no"
                # (19 unchanged elements hidden)
            }
        }

        # (3 unchanged blocks hidden)
    }

  # module.lambdas.aws_lambda_permission.budget_enforcer_reconcile will be created
  + resource "aws_lambda_permission" "budget_enforcer_reconcile" {
      + action              = "lambda:InvokeFunction"
      + function_name       = "javabin-budget-enforcer"
      + id                  = (known after apply)
      + principal           = "events.amazonaws.com"
      + source_arn          = (known after apply)
      + statement_id        = "AllowEventBridgeReconcile"
      + statement_id_prefix = (known after apply)
    }

  # module.lambdas.aws_lambda_permission.team_provisioner_access_schedule will be created
  + resource "aws_lambda_permission" "team_provisioner_access_schedule" {
      + action              = "lambda:InvokeFunction"
      + function_name       = "javabin-team-provisioner"
      + id                  = (known after apply)
      + principal           = "events.amazonaws.com"
      + source_arn          = (known after apply)
      + statement_id        = "AllowEventBridgeAccessSchedule"
      + statement_id_prefix = (known after apply)
    }

Plan: 6 to add, 11 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

LLM Review

Risk: 🔴 HIGH

Plan adds 6 resources (EventBridge rules, targets, Lambda permissions) and updates 11 in place (CI OIDC trust policies, ECS deploy and Lambda IAM policies, Lambda code and env vars); nothing is destroyed, but the CI trust policy changes need explicit sign-off.

  • 🔒 [security] CI OIDC trust policies for ci_infra, ci_infra_plan and ci_registry gain new subject claims: 'repo:javaBin@95047/platform@1175354559:' and 'repo:javaBin@95047/registry@1175355770:' (plus ref/environment variants). The '@' form uses repository-ID-qualified subjects that match a renamed or recreated repo under a different owner/ID. Any repo that holds that ID path could assume these roles, so this should be verified as intentional and the broader wildcard for ci_infra_plan (repo:...:*) reviewed. Those trust policies control production infrastructure, so this change is HIGH-risk regardless of the diff size.
  • 🔒 [security] ci_infra_plan trust condition changes from a single 'repo:javaBin/platform:*' to a list that also includes the ID-qualified wildcard, which broadens who can assume the plan role. Pair with the above review.
  • 🔒 [security] ECS deploy policies for cupcake, javazoneweb, jzwatch and testteam add ecs:TagResource (only on RegisterTaskDefinition), ecs:ListTasks/ecs:ListTagsForResource/ecs:DescribeTasks, and Resource='*' for task-definition actions. Modest, but these grant wide task-definition management across the account.
  • 🔒 [security] budget_enforcer Lambda role gains rds:StopDBInstance/StartDBInstance on all DB instances conditioned only on the presence of an aws:ResourceTag/team tag, plus ssm read/write/delete on /javabin/budget-suspensions. Any RDS instance tagged with a team can be stopped by this role; confirm the tag scoping is acceptable. team_provisioner gains sso:ListAccountAssignments and iam:GetPolicy on team guardrail policies (read-only).
  • 💰 [cost] Two EventBridge schedules are added (budget enforcer every 6 hours, team provisioner access sync every 3 hours) invoking Lambdas. Cost is negligible, but the budget enforcer can stop RDS instances, so its schedule has operational impact.
  • ✅ [routine] Lambda code and environment changes: budget_enforcer and team_provisioner source code hashes change, with new env vars AWS_ACCOUNT_ID, SUSPENSION_PATH and AWS_ACCESS_GROUP=javabin-developer@java.no. No destruction; both functions are updated in place.

@Alexanderamiri
Alexanderamiri merged commit 2df06cd into main Oct 31, 2026
4 checks passed
@Alexanderamiri
Alexanderamiri deleted the fix/budget-suspend-sticks branch October 31, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant