[Alexanderamiri]

Problem

Three findings from the developer-experience audit, all in the team provisioner:

1. New teams got no AWS access. _ensure_team_permission_set stops before the inline policy and the account assignment while javabin-team-<team>-guardrails doesn't exist yet. Platform CI only creates that policy when it applies on the registry's teams-changed dispatch, which runs after the provisioner, and nothing ran the provisioner again. Registry run 40515556958 (javazoneweb) ended at "guardrails": "pending: policy not created yet". The assignment only appeared after someone ran the provisioner by hand on 29 Oct.
2. Members missing from Identity Center were skipped without any notice. Both IC sync paths logged "not in Identity Center — skipping" at INFO and still reported the full member_count. I checked this live: hakon.vadstein@java.no (javazoneweb) has no user in identity store d-9967444724. That store has 24 users, all CreatedBy: SCIM/..., and the newest was created on 2026-05-27.
3. GitHub role default didn't match the docs. The GitHub team role defaulted to member. javabin register-team never writes role:, but the getting-started doc and the registry README say members become maintainers. Adding a repo to a team needs a maintainer.

Change

• New sync_team_access action. Platform CI queues it after every apply (scripts/sync-team-access.sh, async and non-blocking, in the same style as reconcile-app-auth.sh).
  • It covers every team-* Identity Center group whose guardrails policy exists and is tagged as made by platform Terraform. For each one it attaches the guardrails, writes the inline policy, makes the account assignment and re-provisions the permission set.
  • Teams without such a policy (e.g. platform) are left untouched, and the permission set is not even created for them.
  • It posts to #javabin-infra-alerts only when a team gets access for the first time or something fails.
  • Because it runs on every apply, it also rolls inline-policy changes out to existing teams. Until now those only reached a team on its next registry run.
  • No IAM change is needed: javabin-ci-infra can invoke Lambdas, and the provisioner role already has the SSO, identitystore and IAM read permissions.
• Reporting missing Identity Center users. Both IC sync paths now return not_in_identity_center: [emails]. member_count counts only members who are actually in the group, and the log line is now WARNING.
  • Each team's line in the Slack summary flags these members.
  • A new summary block lists every affected person with their groups and says what to do: turn on the AWS IAM Identity Center app under User access in Google Admin.
  • When guardrails are still pending, the team line now says so too.
• Creating the missing users ourselves. Investigated and deliberately not done; the reasons are in docs/lambda-functions.md:
  • A user created through identitystore:CreateUser has no link to the Google account, so suspending the person in Google would not remove their AWS access.
  • Google's later SCIM create for the same UserName would conflict with that user and fail.
• GitHub role default. Members without role: now default to GitHub maintainer (GITHUB_DEFAULT_TEAM_ROLE); role: member opts someone out. An invalid role still falls back to member. The Google group keeps MEMBER as its default. Existing members without role: will be promoted on their team's next registry sync.
• Docs. Updated docs/lambda-functions.md (new GitHub-team and team-AWS-access sections, corrected trigger), docs/ci-workflow.md (post-apply steps, apply triggers), CLAUDE.md (scripts table, alert routing) and the lambdas/main.tf comment.

Verification

• python3 -m unittest discover -s terraform/lambda-src/tests: 232 tests OK. The new test_team_provisioner_access.py covers:
  • team discovery and pagination, and skipping pending teams
  • Slack only on new access or failure, and the teams filter
  • not_in_identity_center in both sync paths, the Slack summary block and the pending line
  • GitHub default, explicit and promoted roles
• python3 -m unittest discover -s scripts/tests: 133 tests OK.
• terraform fmt -check -recursive terraform/platform and terraform validate both pass.
• Live, read-only checks:
  • The IC groups are team-{cupcake,javazoneweb,jzwatch,platform,testteam} plus org groups.
  • Guardrails policies exist for cupcake, javazoneweb, jzwatch and testteam, but not platform.
  • Every IC user is SCIM/...-created.
  • The provisioner's timeout is 300 s.

Follow-ups (not in this PR)

• Google SCIM hasn't created a user since 2026-05-27. That fits users being outside the AWS app's provisioning scope, but it could also mean an expired SCIM access token (they expire after a year). An admin should check IAM Identity Center > Settings > Automatic provisioning and the Google app's provisioning status.
• registry/README.md (which says both "all members are maintainers" and "default: member") and docs/getting-started.md are in other repos and handled separately.

🤖 Generated with Claude Code

[github-actions]

Terraform Plan

🚧 Changes detected — Plan: 0 to add, 1 to change, 0 to destroy.

Plan output

Acquiring state lock. This may take a few moments...
aws_ssm_parameter.atlassian_api_key: Refreshing state... [id=/javabin/platform/atlassian-api-key]
aws_ssm_parameter.atlassian_org_id: Refreshing state... [id=/javabin/platform/atlassian-org-id]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.lambdas.aws_lambda_function.team_provisioner will be updated in-place
  ~ resource "aws_lambda_function" "team_provisioner" {
        id                             = "javabin-team-provisioner"
      ~ last_modified                  = "2026-10-26T14:09:21.000+0000" -> (known after apply)
      ~ source_code_hash               = "i+0n0A3gn0SSOJWZx1C2Aq4/eSSczvrK8hXWYOqCpqs=" -> "oFl7VA29J43xunIno2mriQOb0VL+YMxOo6QwIxqPQGs="
        tags                           = {}
        # (21 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

---

LLM Review

Risk: 🟢 LOW

Plan updates one existing Lambda function (javabin-team-provisioner) in place with a new deployment package (source_code_hash change); no resources created or destroyed.

• ✅ [routine] In-place update of aws_lambda_function.team_provisioner in module.lambdas: only source_code_hash changes (new deployment package), plus last_modified which is computed on apply. This is a routine code deployment.
• 💥 [destruction] Plan summary is 0 to add, 1 to change, 0 to destroy. No replacement or deletion of any resource is shown, so no data loss risk from this plan.
• 🔒 [security] The diff shows no changes to the function's IAM execution role, environment variables, VPC/security group configuration, or resource-based policies. No new public access is introduced. Note that the visible diff does not show the code contents, so the new package itself should be verified to come from the expected source/commit.
• 💰 [cost] No new billable resources (NAT gateways, RDS, load balancers, etc.) are created. Cost impact is negligible.

[Alexanderamiri]

Review follow-up (d5137c3):

Accepted: sync_team_access would announce every team on every apply. I confirmed it. CloudWatch /aws/lambda/javabin-team-provisioner logs "Assigned javabin-team-cupcake to group c3e40842-..." three times in the last 30 days and never logs the Conflict branch. AWS accepts a create for an assignment that already exists without raising an error, so already_existed was never set.

Changes:

• New _assignment_exists() uses sso:ListAccountAssignments (paginated, filtered to the group). _create_account_assignment skips the create when the assignment exists and returns already_existed: True. It sets already_existed: False only when the lookup confirmed the assignment is new. If the lookup fails, the key is left out.
• _noteworthy now announces only when already_existed is False, so a failed lookup no longer posts a Slack message. Errors and failed provisioning are still announced. Re-provisioning still runs for existing assignments, so inline policy changes keep rolling out.
• IAM: added sso:ListAccountAssignments to the provisioner role (terraform/platform/lambdas/main.tf). It ships in the same apply as the code.
• Tests: SyncTeamAccessTest now mocks the SSO client the way AWS behaves (a create on an existing assignment succeeds), not _ensure_team_permission_set. New cases cover an existing assignment (no create, no Slack), a failed lookup (no Slack) and a new assignment (announced). With the old handler two of these tests fail; with the new one they pass.
• Docs: docs/lambda-functions.md explains how the handler tells new assignments from existing ones.

Verified with python3 -m unittest discover -s terraform/lambda-src/tests (233 OK), terraform fmt -check and terraform validate in terraform/platform.

Rejected: none.

[github-actions]

Terraform Plan

🚧 Changes detected — Plan: 0 to add, 2 to change, 0 to destroy.

Plan output

Acquiring state lock. This may take a few moments...
aws_ssm_parameter.atlassian_api_key: Refreshing state... [id=/javabin/platform/atlassian-api-key]
aws_ssm_parameter.atlassian_org_id: Refreshing state... [id=/javabin/platform/atlassian-org-id]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.lambdas.aws_iam_role_policy.team_provisioner will be updated in-place
  ~ resource "aws_iam_role_policy" "team_provisioner" {
        id     = "javabin-team-provisioner:javabin-team-provisioner"
        name   = "javabin-team-provisioner"
      ~ policy = jsonencode(
          ~ {
              ~ Statement = [
                    # (4 unchanged elements hidden)
                    {
                        Action   = [
                            "identitystore:CreateGroup",
                            "identitystore:DescribeGroup",
                            "identitystore:DescribeUser",
                            "identitystore:ListGroups",
                            "identitystore:CreateGroupMembership",
                            "identitystore:ListGroupMemberships",
                            "identitystore:DeleteGroupMembership",
                            "identitystore:ListUsers",
                        ]
                        Effect   = "Allow"
                        Resource = "*"
                        Sid      = "IdentityStoreSync"
                    },
                  ~ {
                      ~ Action   = [
                            # (1 unchanged element hidden)
                            "sso:DescribeAccountAssignmentCreationStatus",
                          + "sso:ListAccountAssignments",
                            "sso:ListPermissionSets",
                            # (7 unchanged elements hidden)
                        ]
                        # (3 unchanged attributes hidden)
                    },
                    {
                        Action   = [
                            "iam:GetPolicy",
                            "iam:GetPolicyVersion",
                            "iam:ListPolicyTags",
                        ]
                        Effect   = "Allow"
                        Resource = "arn:aws:iam::553637109631:policy/javabin-team-*-guardrails"
                        Sid      = "ReadTeamGuardrails"
                    },
                    # (1 unchanged element hidden)
                ]
                # (1 unchanged attribute hidden)
            }
        )
        # (1 unchanged attribute hidden)
    }

  # module.lambdas.aws_lambda_function.team_provisioner will be updated in-place
  ~ resource "aws_lambda_function" "team_provisioner" {
        id                             = "javabin-team-provisioner"
      ~ last_modified                  = "2026-10-26T14:09:21.000+0000" -> (known after apply)
      ~ source_code_hash               = "i+0n0A3gn0SSOJWZx1C2Aq4/eSSczvrK8hXWYOqCpqs=" -> "cH9xoIvkYZPX3I5j9mS8AD1QKKWJ4Q7umcXH3n+damY="
        tags                           = {}
        # (21 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

---

LLM Review

Risk: 🟢 LOW

Plan updates two existing resources in place (0 added, 0 destroyed): the team provisioner Lambda's IAM role policy gains sso:ListAccountAssignments, and the Lambda code is redeployed with a new source hash.

• 🔒 [security] The IAM role policy for javabin-team-provisioner adds "sso:ListAccountAssignments" to an existing statement with Effect = "Allow" (Resource is not shown as changed, so it inherits that statement's existing scope). This is a read-only SSO action that lets the provisioner list which accounts/permission sets are assigned. It is a minor widening of read access, not a new write or public exposure. Worth confirming the statement's Resource scope is tight.
• 🔒 [security] The other policy statements (IdentityStore sync with Resource = "", and read-only guardrail IAM access scoped to javabin-team--guardrails) are unchanged in this plan. No security group, public access, or permission boundary changes.
• ✅ [routine] Lambda function javabin-team-provisioner has a new source_code_hash, meaning new deployment code is being shipped. Code diff is not visible in the plan, so review the artifact source in version control before apply.
• 💥 [destruction] No resources are destroyed or replaced. Both changes are in-place updates.
• 💰 [cost] No new billable resources are created (no NAT gateways, RDS, load balancers). Cost impact is negligible.

[Alexanderamiri]

Team and kodesmia members now get AWS access (017185b)

Why. Google's SCIM provisioning creates Identity Center users only for direct members of drift@, styret@ and javabin-developer@java.no. It does not expand nested groups. javabin-developer@ has one member (alexander.amiri), so team members and most of kodesmia never got a user, and the provisioner skipped them with "not in Identity Center".

Change

1. Team sync.
   • Each member's java.no address (taken from google) becomes a direct MEMBER of AWS_ACCESS_GROUP. The default is javabin-developer@java.no, set by Terraform var.aws_access_group.
   • It only adds. The result lists added_members, plus preserved for members who are in neither kodesmia nor a team (they stay), and skipped_external for personal addresses.
2. flatten: true on access groups. The role groups' individual members are added directly, taken from the resolved members already in the sync_groups_and_heros payload, instead of the nested group email. Additive, with the same reporting. For kodesmia, the registry needs this entry in groups/access.yaml (registry repo, not in this PR):

   - name: javabin-developer
     google: javabin-developer@java.no
     description: Google SCIM provisions direct members to IAM Identity Center
     groups: [kodesmia]
     flatten: true

3. Ordering and retry. In this PR, sync_team_access runs right after the platform apply, so SCIM hasn't created the new users by then, and before this commit nothing ran later.
   • New action sync_access_members on an EventBridge schedule, javabin-team-provisioner-access-schedule, rate(3 hours).
   • It reads the registry members the provisioner already records in the memberships table (team-* and cognito-enabled groups such as kodesmia).
   • It adds recorded team members to javabin-developer@, then adds members whose Identity Center user now exists to the Identity Center group with the same name. It never removes anyone.
   • It posts to Slack only when it added someone or an add failed. No new IAM permissions are needed (Scan, ListUsers and CreateGroupMembership were already granted).
4. Skipped-member report. It is still in Slack and now also in the response body as not_in_identity_center: {email: reason}. The possible reasons are:
   • pending SCIM: just added to javabin-developer@java.no
   • pending SCIM: already a direct member of a provisioned group
   • can't be provisioned: personal address / suspended in Google / not in Google Workspace
   • not a direct member of javabin-developer@java.no

Live dry run (read-only)

I imported the handler locally with the provisioner's own credentials (_get_google_access_token, SSM google-admin-sa/google-admin-email). Every non-GET Google call and every Identity Center create/delete_group_membership was monkeypatched to record instead of send. Input was registry origin/main: 5 teams plus the kodesmia flatten entry above.

Would add to javabin-developer@java.no (22):
alexander.samsig, anders.karlsen, caroline.vannebo, chris.searle, dennis.knudsen, dervis.mansuroglu, ellaoui, emil.nilsen, erlend.hamnaberg, hakon.vadstein, henrik.eide, nicolas.nordhagen, numan.uskudar, oyvind.lokling, rina.andriana, runhild.heggem, sandra.lekve, stian.nygaard, stig.lau, tanet.trimas, trym.froystein, vytautas.zaleckas (all @java.no)

• From team sync alone: chris.searle, hakon.vadstein, stian.nygaard, vytautas.zaleckas.
• Nothing preserved, no personal addresses skipped, no failures.

kodesmia and team members not in Identity Center today (12). All are classified pending SCIM: just added to javabin-developer@java.no. None are suspended, missing from Google or personal:
caroline.vannebo, ellaoui, emil.nilsen, erlend.hamnaberg, hakon.vadstein, henrik.eide, nicolas.nordhagen, numan.uskudar, rina.andriana, runhild.heggem, stig.lau, trym.froystein

The other 10 already have an Identity Center user through drift@ or styret@. Adding them to javabin-developer@ changes nothing for SCIM.

Scheduled retry today: no Identity Center writes. Everyone missing is still waiting for a user: the 12 above in kodesmia, hakon in team-javazoneweb, and erlend.bergheim@java.no in drift. That last one is not caused by this PR: the registry has erlend.bergheim@java.no, but Identity Center and Google's drift@ have erlend.klakegg.bergheim@java.no.

Verification

• python3 -m unittest discover -s terraform/lambda-src/tests: 246 OK. The new tests are in test_team_provisioner_aws_access.py. They cover team adds, flatten vs nested, preserved/skipped/failed members, every reason, the handler and group-sync bodies, and the schedule's retry, quiet path and team adds.
• python3 -m unittest discover -s scripts/tests: OK.
• terraform fmt -check -recursive terraform/platform and terraform validate: OK.

Docs

• docs/lambda-functions.md has a new section, "How a member gets AWS access", and its trigger line now mentions the schedule.
• CLAUDE.md has the flow lines and a gotcha. lambdas/README.md has the updated trigger.
• The GitHub role default was already documented here as maintainer, with role: member as the opt-out.

Not done here (outside this worktree)

• docs/getting-started.md is in javaBin/docs (PR fix(docs): match auth and getting-started to the platform code docs#5). It should say that registry team members default to GitHub maintainer, that role: member restricts that, and how AWS access works: registry adds you to javabin-developer@, then Google SCIM creates your Identity Center user (can take a while), then the provisioner adds you to team-<team> within ~3 hours, then run javabin aws setup.
• The registry's access.yaml entry shown above.
• registry/README.md, which contradicts itself on the role default.
• Nothing writes to Google until this is applied and the registry change merges. The first adds will come from the next team or group sync, or from the first scheduled run (team members only).

🤖 Generated with Claude Code

[github-actions]

Terraform Plan

🚧 Changes detected — Plan: 3 to add, 2 to change, 0 to destroy.

Plan output

Acquiring state lock. This may take a few moments...
aws_ssm_parameter.atlassian_org_id: Refreshing state... [id=/javabin/platform/atlassian-org-id]
aws_ssm_parameter.atlassian_api_key: Refreshing state... [id=/javabin/platform/atlassian-api-key]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # module.lambdas.aws_cloudwatch_event_rule.team_provisioner_access_schedule will be created
  + resource "aws_cloudwatch_event_rule" "team_provisioner_access_schedule" {
      + arn                 = (known after apply)
      + description         = "Retry Identity Center group memberships for members Google SCIM has since provisioned"
      + event_bus_name      = "default"
      + force_destroy       = false
      + id                  = (known after apply)
      + name                = "javabin-team-provisioner-access-schedule"
      + name_prefix         = (known after apply)
      + schedule_expression = "rate(3 hours)"
      + tags_all            = {
          + "environment" = "production"
          + "managed-by"  = "terraform"
          + "repo"        = "javaBin/platform"
          + "service"     = "platform"
          + "team"        = "platform"
        }
    }

  # module.lambdas.aws_cloudwatch_event_target.team_provisioner_access will be created
  + resource "aws_cloudwatch_event_target" "team_provisioner_access" {
      + arn            = "arn:aws:lambda:eu-central-1:553637109631:function:javabin-team-provisioner"
      + event_bus_name = "default"
      + force_destroy  = false
      + id             = (known after apply)
      + input          = jsonencode(
            {
              + action = "sync_access_members"
            }
        )
      + rule           = "javabin-team-provisioner-access-schedule"
      + target_id      = "invoke-team-provisioner-access"
    }

  # module.lambdas.aws_iam_role_policy.team_provisioner will be updated in-place
  ~ resource "aws_iam_role_policy" "team_provisioner" {
        id     = "javabin-team-provisioner:javabin-team-provisioner"
        name   = "javabin-team-provisioner"
      ~ policy = jsonencode(
          ~ {
              ~ Statement = [
                    # (4 unchanged elements hidden)
                    {
                        Action   = [
                            "identitystore:CreateGroup",
                            "identitystore:DescribeGroup",
                            "identitystore:DescribeUser",
                            "identitystore:ListGroups",
                            "identitystore:CreateGroupMembership",
                            "identitystore:ListGroupMemberships",
                            "identitystore:DeleteGroupMembership",
                            "identitystore:ListUsers",
                        ]
                        Effect   = "Allow"
                        Resource = "*"
                        Sid      = "IdentityStoreSync"
                    },
                  ~ {
                      ~ Action   = [
                            # (1 unchanged element hidden)
                            "sso:DescribeAccountAssignmentCreationStatus",
                          + "sso:ListAccountAssignments",
                            "sso:ListPermissionSets",
                            # (7 unchanged elements hidden)
                        ]
                        # (3 unchanged attributes hidden)
                    },
                    {
                        Action   = [
                            "iam:GetPolicy",
                            "iam:GetPolicyVersion",
                            "iam:ListPolicyTags",
                        ]
                        Effect   = "Allow"
                        Resource = "arn:aws:iam::553637109631:policy/javabin-team-*-guardrails"
                        Sid      = "ReadTeamGuardrails"
                    },
                    # (1 unchanged element hidden)
                ]
                # (1 unchanged attribute hidden)
            }
        )
        # (1 unchanged attribute hidden)
    }

  # module.lambdas.aws_lambda_function.team_provisioner will be updated in-place
  ~ resource "aws_lambda_function" "team_provisioner" {
        id                             = "javabin-team-provisioner"
      ~ last_modified                  = "2026-10-26T14:09:21.000+0000" -> (known after apply)
      ~ source_code_hash               = "i+0n0A3gn0SSOJWZx1C2Aq4/eSSczvrK8hXWYOqCpqs=" -> "NhpsEZ7z4TSfn+HHPW1O5VfUrAWq0B92kMB9qd3+NgE="
        tags                           = {}
        # (21 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              + "AWS_ACCESS_GROUP"             = "javabin-developer@java.no"
                # (19 unchanged elements hidden)
            }
        }

        # (3 unchanged blocks hidden)
    }

  # module.lambdas.aws_lambda_permission.team_provisioner_access_schedule will be created
  + resource "aws_lambda_permission" "team_provisioner_access_schedule" {
      + action              = "lambda:InvokeFunction"
      + function_name       = "javabin-team-provisioner"
      + id                  = (known after apply)
      + principal           = "events.amazonaws.com"
      + source_arn          = (known after apply)
      + statement_id        = "AllowEventBridgeAccessSchedule"
      + statement_id_prefix = (known after apply)
    }

Plan: 3 to add, 2 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

---

LLM Review

Risk: 🟢 LOW

Plan adds an EventBridge schedule (every 3 hours) that invokes the team-provisioner Lambda to retry Identity Center group memberships, plus a Lambda permission for it, and updates the Lambda code, its environment, and its IAM policy; 0 destroys.

• 🔒 [security] IAM role policy adds sso:ListAccountAssignments to the existing Allow statement; this is a read-only action and is a minor widening of read scope on the team-provisioner role.
• 🔒 [security] The new Lambda permission is scoped to events.amazonaws.com with a statement ID, and the EventBridge rule targets only the team-provisioner function, so invocation access is not broadened to any other principal.
• ✅ [routine] Lambda code is updated in place (source_code_hash change) and a new AWS_ACCESS_GROUP environment variable (javabin-developer@java.no) is added; no replacement or destruction of the function.
• 💰 [cost] EventBridge rule running every 3 hours (about 240 invocations/month) with Lambda runtime cost, negligible in practice.
• ✅ [routine] The plan has 0 destroys; no databases, security groups, or permission boundaries are changed, so no high-risk items are present.