Skip to content

Adopt the role controller on a live fleet and set its defaults #184

Closed

Description

@Mearman

The last piece of the dynamic regrading epic: decide the collection's defaults and bring the role controller into use on a real fleet.

Defaults. The controller stays off unless github_runner_cluster_role_controller is true, and dry-run unless FLEET_ROLE_DRY_RUN is set false, so a consumer that upgrades changes nothing. No host is eligible or pinned by default. The target voter count defaults to 3. The README's server selection section is rewritten for the case where the controller manages roles: inventory order gives the first role only.

Memory-pressure regrading (#140). The controller refuses to change a role while any voter is below full, and uses the regrade core's own memory signals, so regrading and role changes share one Lease and never step together. That makes regrading a prerequisite on every eligible server, which strengthens the case in #140 for making it the default for a server; the trial #140 asks for can include native servers once the systemd actuator has shipped. #140 stays its own decision.

Adoption on a live fleet. In order:

  1. Mark the eligible hosts and stage every native one as a standby server with the service left stopped (the role's native start flag off), so certificates and configuration exist before any promotion. Every server's certificate gains the eligible hosts' names, which takes one rolling restart of the servers, one at a time through the role's quorum guard.
  2. Run the probe and the controller in dry-run for at least one window on the live cluster, read fleet-role-status against hand readings on each host, and correct any probe that disagrees before going on.
  3. Do the first swap by hand through the controller's command while the owner watches, with a fresh snapshot.
  4. Turn dry-run off.

A fleet that has decided particular servers stay voters whatever their link keeps that decision with github_runner_cluster_voter_pinned; dynamic regrading otherwise moves votes off a host whose path is poor, which is its purpose, so each pin is set in the fleet's inventory with a comment saying why.

Tests: the fleet's own --check run before each step, and the controller's dry-run output read against the hosts. No collection change here beyond docs and defaults, so its CI is the existing suites.

Activity

  1. added theissue type on Nov 3, 2026
  2. Mearman commented on Nov 7, 2026

    @Mearman
    MemberAuthor

    Closing: the controller is adopted on the live fleet in dry run, and no swap is needed now. Steps, against the order in this issue:

    1. Eligible hosts are marked and staged as standby servers with the service stopped: asus, i5 and air carry the eligibility label (the gate's D4 reads the staged unit and an active agent on asus and i5).
    2. The probe and the controller ran in dry run on the live cluster and were read against hand measurements: the i5's fdatasync 99th percentile was 11.99 ms in the status against 10.2 ms by hand with fio, and a TCP connect to m1 on the etcd peer port measured 38 ms at the 99th percentile with the probe's own curl method against 94 ms in the controller's 25 minute window. No probe needed correcting.
    3. The first manual swap was not run, and step 4 (turning dry run off) was not done: the voters read fit and the status says "no voter is unfit", so there is nothing to swap, and the only host that ever ranked as a candidate (the asus) is cordoned with failing memory. Dry run stays the default.

    A defect found on the way is fixed and released in 2.44.2: a cordoned standby was ranked as a candidate (#239). Reopen this when a voter is judged unfit and a swap is wanted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions