Skip to content

Add reviewed TCP route treatments, controller, and VM soak tests - #282

Merged
randomizedcoder merged 28 commits into
mainfrom
feat/tcp-route-controller
Nov 14, 2026
Merged

randomizedcoder merged 28 commits into
mainfrom
feat/tcp-route-controller

Conversation

@randomizedcoder

@randomizedcoder randomizedcoder commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Adds an opt-in TCP route-treatment control plane: ClickHouse socket evidence produces destination-specific hypotheses, operators approve gradual per-location canaries, and agents apply and withdraw verified route overrides through goip. xtcp2 stays observe-only unless this is deployed; route application is disabled by default and needs explicit agent configuration.

The nearby-traffic profile requests CUBIC, a 5 ms minimum RTO, quick ACK, and initial send and receive windows of 100 segments.

The branch is the original feature commit followed by eighteen commits that apply an architecture review item by item, so the history reads as the review being addressed, then eight follow-up commits (F1-F5) for the review's TODOs. docs/status-tcp-controller.md maps each review item to its commit, records the design notes per item, and lists the open TODOs.

Shape

  • pkg/treatment is the shared contract only: domain types, IDs, limits and validators. No I/O, no gRPC, no SQL.
  • pkg/treatment/identity parses SPIFFE identities, builds the mTLS config, and provides the unary and stream interceptors that resolve the caller once and enforce the role each RPC declares as a proto option. A test fails when any registered RPC lacks a role.
  • pkg/treatment/experiment is the pure decision core (Evaluate, Advance, cohort allocation) plus an explicit (status, event) transition table; every pair has a row, and outcomes are an Outcome enum rather than strings.
  • pkg/treatment/reconcile holds the reconciler, the journal, the Backend interface and a Mode type. The journal is a journal-specific prototext message, so neither the RPC proto nor the domain model is an on-disk format. The journal lock waits, bounded by the caller's context; only xtcp2-treatmentctl -mode=status stays non-blocking.
  • pkg/treatment/agent is a library used two ways. cmd/xtcp2-treatment-agent runs it standalone. cmd/xtcp2 embeds it behind the treatment build tag (-treatmentConfig). The untagged daemon reads the standalone agent's journal snapshot through pkg/treatment/attribution (-treatmentAttribution) and links no gRPC credentials or TLS. The watchdog (xtcp2-treatmentctl) is a separate process in both modes and can only withdraw.
  • One location rule: the daemon and its agent refuse to start when their locations differ, a daemon without a location adopts the agent's, and records are no longer rewritten per host.
  • pkg/treatment/controller is thin gRPC handlers and orchestration (Maintain, Discover) over ports. controller/store defines the repository interfaces (Agents, Experiments, Evidence, Holds, Audit, Metrics) and a Tx boundary that takes the mutation lock. store/postgres holds every SQL string, on a relational schema, through pgx v5 via database/sql (lib/pq removed). store/memory backs the handler tests and store/storetest is the conformance suite both stores pass. Evidence is stored once with a content digest; audit payloads are the only JSONB.
  • controller/evidence defines the Source and CandidateSource ports; ClickHouse is one adapter implementing both.
  • Evidence is bound to sessions: each agent process draws a collection nonce, writes it into its journal header (where a telemetry-only daemon stamps it into records) and reports it with its boot ID at check-in. The controller keeps every (agent_id, boot_id, collection_id) it has heard, at most eight per agent per window, and the analysis query measures only records stamped with one of them; the rest are counted per host as unbound_sockets and shown to the reviewer. Discovery proposes only enrolled hosts.
  • Experiments.List and Get load in a fixed number of statements (six for any list size), pinned by a statement-counting test.
  • The proto lives in proto/xtcp_treatment/v1 (treatment.proto, agent.proto, operator.proto) with buf.validate rules that mirror the Go validators and a per-RPC role option. It is wire-only; pkg/treatment/wire converts at the gRPC boundary and a test checks the proto rules against the validators.
  • internal/goip/actuate holds the route backend, built on a typed xtcpnl.RuleWrite builder parallel to RouteWrite; the argv parser is out of the mutation path.
  • Scale: the notification scan is a watermark over desired_updated_at stamped with the store clock, check-ins rewrite algorithm and record rows only when the report digest changes, and Maintain and Discover run under a housekeeping lease so one replica does periodic work while the others skip.
  • TestTreatmentImportBoundaries enforces the layering with go list -deps: the untagged daemon links no database/sql, pgx, ClickHouse or controller code; the reconciler and watchdog link no gRPC credentials; identity is a leaf.
  • Socket protobuf, ClickHouse and Parquet records carry treatment attribution and the observed congestion algorithm, stamped through a nil-checked, zero-allocation hook.
  • Nix: seven-guest VM integration test with Go orchestration (repeated netem RTT changes, discovery, reviewed canaries, rollback, cooldown), unit and race harnesses with PostgreSQL and ClickHouse fixtures, benchmark target, NixOS module and package wiring for the four binaries.

Docs

  • docs/tcp-controller.md: deployment, rollout, recovery, database, trust boundaries and threat model (what a host can and cannot assert about evidence, and how the analysis query, sample minimums, freshness, human review and immutable evidence bound it), names and surfaces, and SIGHUP meaning "reconcile now" only.
  • docs/tcp-controller-testing.md: test contracts, a coverage table generated by the unit target, and remaining gaps.
  • docs/benchmarks/tcp-controller/postgres-indexes.txt: produced by nix build .#bench-tcp-controller-postgres-indexes.
  • docs/status-tcp-controller.md: item-to-commit table, per-item notes, and open TODOs (gRPC stubs still reachable through the flat-record package, evidence window parameters approaching ClickHouse's URL limit at 10,000 agents, ipmeta-bootstrap -help exit code, regenerating every proto with the pinned generators).
  • docs/README.md now states that xtcp2 changes nothing on the host except through this opt-in, human-reviewed controller.

Validation

All on the branch head:

  • nix build .#checks.x86_64-linux.lint-baseline: pass (all three golangci tiers, zero findings). The comprehensive tier's 107 findings on the new code were fixed by restructuring, with no //nolint, exclusions or threshold changes.
  • nix build .#test-tcp-controller-unit: pass (PostgreSQL repository conformance for every method, handler tests on the in-memory store, ClickHouse SQL counter-delta test, import-boundary test, state-machine and wire round-trip tables).
  • nix build .#test-tcp-controller-race: pass.
  • nix run .#test-tcp-controller-vm -- --accel=auto: pass, all 8 sentinels (netem RTT change and discovery, reviewed canary, controller restart persistence, holds, evidence binding, harmful-treatment rollback, watchdog expiry, overall). In the binding stage, records copied under a fake boot and nonce for one cohort host came back as 32 unbound sockets while its 32 bound sockets were measured as before.
  • nix build .#checks.x86_64-linux.cli-help-smoke-<binary>: pass for the four treatment binaries.
  • nix build .#bench-tcp-controller-postgres-indexes: pass.
  • go test -tags treatment ./cmd/xtcp2/: pass (embedded agent mode).

Operational limits

Promotion and stable-treatment replacement require operator review. The actuator supports host-namespace IPv4 /32 and IPv6 /128 overrides with simple baseline routing; established sockets can retain previous settings. Route conflicts are refused, and disabling the owned lookup after an apply failure can itself fail. Discovery examines its ten busiest candidates before filtering existing treatments, which can defer other candidates. The schema is pre-release: tables and indexes are created by Migrate with no ALTER path yet. Evidence is bound to sessions the agents report over mTLS, but the nonce travels in the same unauthenticated telemetry, so a reader of the topic can copy a live session; see the threat model section for what that does and does not prove.

🤖 Generated with Claude Code

randomizedcoder and others added 28 commits October 25, 2026 15:54
…ring

The agent imported pkg/treatment/controller for TLSConfig and
CertificateIdentity, which pulled database/sql, lib/pq, the ClickHouse
client and both gRPC server implementations into every xtcp2 build
flavor. Move SPIFFE identity parsing, peer resolution and TLS config
into pkg/treatment/identity, a leaf package both sides can share, and
repoint the agent, controller handlers, status page, both binaries and
the integration tool.

Add pkg/treatment/boundaries_test.go, a parser-walk layering test in
the style of internal/goip/imports_test.go. It asserts that cmd/xtcp2
and pkg/treatment/agent never reach the controller, a SQL driver or
ClickHouse, that the controller never imports the agent, and records
two rows to tighten once the contract package stops importing the
proto and the watchdog builds on the reconcile package.

The identity table test moves into the new package and gains boundary
and corner rows (128-byte IDs, escaped separators, scheme casing).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
pkg/treatment mixed three audiences: the shared contract (validators,
limits), controller-only statistics (Evaluate, Advance) and host-side
actuation (Reconciler, Journal). The controller imported a Backend
interface it never used and the agent imported the t-distribution code
it never ran, and nothing stopped either from growing into the other.

Move reconcile.go and journal.go to pkg/treatment/reconcile and
experiment.go to pkg/treatment/experiment; the root keeps only the
contract. In reconcile, replace the expiryOnly bool with a Mode type
(ModeFull, ModeExpiryOnly) that Run rejects when out of range, and make
BeginReconcile a declared member of Backend instead of a hidden type
assertion. The store helper named experiment becomes loadExperiment to
free the package name.

Tests move with their code. The desired-state and experiment fixtures
that three packages shared move into pkg/treatment/internal/fixtures,
which imports only generated types so the contract package's own tests
can use it without a cycle. The layering test gains rows for the two
new packages and for "controller never reaches reconcile or goip"; two
rows note the gRPC forbid that becomes possible once domain types
replace the generated package.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
…ion rules

proto/xtcp/controller/v1/controller.proto nested the path unlike every
other proto in the repo, generated a Go package whose name no longer
matched the path, and carried no buf.validate rules: all input checking
was hand-written. Move it to proto/xtcp_treatment/v1 as package
xtcp_treatment.v1, split into treatment.proto (types), agent.proto
(AgentControlService) and operator.proto (OperatorService), generated
as gen/go/xtcp_treatment. The 37 Go importers keep their pb alias.

Every client-controlled field now carries a rule that mirrors the bound
in pkg/treatment/validate.go: identifier pattern, revision and
generation > 0, TCP parameter ranges, host-prefix destinations, evidence
interval ordering and digest shape, policy ranges with the
observation-within-phase relation, cohort size and uniqueness, reason
and request_id limits, exactly-one control selector, and the reserved
".result" request_id suffix. Contextual checks that span messages stay
in Go. A new per-RPC option (xtcp_treatment.v1.role) declares which
certificate class each method requires; enforcement moves into an
interceptor in a later commit.

pkg/treatment/proto_rules_test.go pins the mirror with positive,
negative, boundary and corner rows through protovalidate, and
TestEveryRPCDeclaresARole fails if a method is added without a role.

Generated outputs for the other protos were left untouched: the local
toolchain is one protoc-gen-go patch version newer than what produced
them, and that drift is not this change's business.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Status changes now go through one transition table in
pkg/treatment/experiment/statemachine.go (Transition, Event, Terminal).
Review, Control and Maintain apply named events instead of assigning
statuses inline, and any pair outside the table fails with
FailedPrecondition. TestTransitionExhaustive visits every status and
event pair against an independently written legal table.

Evaluation results move from free strings to the Outcome enum in
treatment.proto, mirrored by experiment.Outcome with Promotable,
Withdraws, RenewsLease, ParseOutcome, Wire and OutcomeFromWire. The
"observing" state the controller set after verified application becomes
OUTCOME_OBSERVING. TestOutcomeMirrorsWire pins the Go and proto values
to each other; unknown wire values are rejected rather than defaulted.

Assignment.id is removed (reserved) in favour of experiment_id, which it
always equalled. docs/tcp-controller.md gains a lifecycle section.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Item 4 of the architecture review (B3, B4): the proto is now wire-only
and PostgreSQL is relational.

Domain and wire
- pkg/treatment holds proto-free Go domain types (Treatment, Experiment,
  Evidence, Assignment, DesiredState, ApplicationRecord, ...).
- pkg/treatment/wire converts domain <-> xtcp_treatment.v1 at the gRPC
  boundary; round-trip tests cover populated, unset-optional and enum
  coverage cases.

Store ports
- pkg/treatment/controller/store defines Agents, Experiments, Evidence,
  Holds, Audit and Metrics repositories, Repos, and Store{Repos, Tx,
  Ping, Close} with ErrNotFound/ErrNotEnrolled/ErrExists/
  ErrEvidenceConflict/ErrRequestConflict sentinels.
- store/postgres: hand-written SQL only, pgx v5 through database/sql,
  relational schema (agents, agent_algorithms, experiments,
  experiment_cohort, phases, assignments, application_records, evidence,
  evidence_hosts, holds, events). Tx takes the advisory lock. Evidence is
  stored once and referenced by ID; the digest hashes timestamps at
  microsecond precision so it agrees with timestamptz storage.
- store/memory: in-memory implementation with snapshot rollback so
  handlers test without a database.
- store/storetest: conformance suite run by both implementations, with
  table-driven rows across positive/negative/boundary/corner cases
  (desired revision fence, cohort immutability, max cohort and
  evidence_hosts 10000, cidr /32 and /128, advisory-lock serialisation,
  request-ID replay with same and different bodies, nanosecond clocks).
- store/pgtest: per-test schema fixture keyed off
  XTCP_CONTROLLER_TEST_DSN, dropped on cleanup.

Controller
- server.go, status.go, maintenance.go, discovery.go no longer contain
  SQL; all mutations go through Store.Tx.
- Experiment.UpdatedAt is stamped by the controller clock on save, so
  discovery cooldown follows the injected clock.
- Handler tests run as <Test>/memory and <Test>/postgres subtests;
  TestPostgresConcurrentReviewFencesReplicas stays Postgres-only for the
  Nix harness grep. The index EXPLAIN test moved to store/postgres.
- cmd/xtcp2-controller opens the store through postgres.Open and
  Migrate.

Boundaries
- lib/pq removed from go.mod; github.com/jackc/pgx/v5 added.
- boundaries_test tightens rules: domain and experiment packages are
  proto-free; the daemon, memory store and identity forbid database/sql,
  pgx and ClickHouse; SQL lives only in store/postgres.

Docs
- docs/tcp-controller.md package table, schema/index names and test
  instructions updated.
- docs/status-tcp-controller.md added with progress and TODOs
  (journal still pb prototext, Experiments.List N+1, Observe rewrites
  records per heartbeat, benchmark refresh).
- nix/versions.nix goVendorHash updated for the pgx dependency.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
…cal and attribution packages

Dual-mode agent, part 1. The host side no longer depends on the RPC proto
package, so the watchdog and the journal readers can build without the
gRPC service stubs that share a Go package with the RPC messages.

Journal
- New proto package xtcp_treatment_journal.v1 (journal.proto), generated
  into gen/go/xtcp_treatment_journal: Journal{agent_id, location, boot_id,
  desired, locally_suspended, suspension_reason, records, updated_at} with
  its own Treatment/Assignment/Record messages. A file format, not the
  model; the pre-rework LocalState text fails closed.
- reconcile.State and reconcile.Record are the host-side types. Record
  embeds treatment.ApplicationRecord plus InstalledRoute/InstalledBootID,
  so State.Reports() cannot leak kernel bytes into a check-in.
- reconcile.MarshalText/UnmarshalText/ReadSnapshot convert and validate;
  every load runs the domain validators and refuses unknown enum values.

Reconciler
- Backend.Plan takes *treatment.Treatment. Reconciler gains Location and
  writes the agent_id/location/boot_id header on every save; a journal
  written by another agent is refused for Run and SuspendLocal.
- treatment.Assignment and DesiredState gain Clone and Equal.

Packages
- pkg/treatment/local: the shared controller-agent.json Config and
  LoadConfig, HostIdentity, Host (reconciler + goip adapter), Protect.
  agent.Config is an alias; agent.LoadConfig delegates.
- pkg/treatment/attribution: Index built from a journal snapshot, Stamp
  on the hot path (one atomic load, one map lookup, no allocation),
  Stamper. The embedded agent publishes an index after every pass and an
  identity-only index before the first; a telemetry-only daemon will feed
  it from the journal directory in part 2.
- pkg/treatment/agent builds on local and attribution; check-ins go
  through wire.Report and wire.ParseDesired.
- cmd/xtcp2-treatmentctl builds on local and reconcile only.

Tests (table-driven, description/expected, all four categories)
- reconcile: journal round trip (populated, unset-optional, empty,
  tombstone-only), enum mirror, limits (max records, over cap, oversized,
  bad destination, bad lease), invalid snapshots (now also unknown mode
  and status, bad timestamp, bad identity, legacy format, garbage),
  lock-free reader under lock and against symlinks and the size limit,
  projections follow the snapshot, header identifies the writer, reports
  strip local fields.
- attribution: stamp contracts (15 rows), identity modes (embedded,
  attribution, identity-only, nil), concurrent publish, benchmarks.
- local: LoadConfig contracts, New contracts, Plan validates.
- agent: lifecycle rows (connected, offline, undecodable desired state),
  stamp before first reconcile.
- boundaries: reconcile forbids the RPC proto, wire, goip and gRPC; the
  watchdog forbids identity, agent, attribution, wire and the RPC proto;
  local and attribution rules added; the controller forbids local,
  attribution and the journal proto.

Docs: package table, boundary sentence and journal format in
docs/tcp-controller.md; D1/D2 rows and notes in docs/status-tcp-controller.md,
with the xtcpnl -> flat record -> gRPC chain recorded as a TODO.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
…d tag, standalone binary

Dual-mode agent, part 2. The daemon now gets treatment provenance one of
two ways, never both:

- -treatmentAttribution <dir> (always compiled): a standalone
  xtcp2-treatment-agent owns the journal; xtcp2 watches the directory
  (inotify on the atomic rename of state.textproto, 30 s poll fallback,
  SIGHUP re-reads) and rebuilds the exposure index from the snapshot. A
  missing or invalid snapshot clears attribution instead of stamping
  stale state. The daemon keeps its own -location and logs once if the
  journal disagrees.
- -treatmentConfig <file> (build tag `treatment`): the agent runs
  in-process, as before. Without the tag the flag is refused at startup
  with both remedies named, the same contract as a slim build asked for
  an enricher it lacks. Both flags exist in every build, so the flag
  golden stays flavor-neutral.

New
- pkg/treatment/attribution.Watcher (journal directory watcher).
- cmd/xtcp2-treatment-agent: the agent library as its own process;
  SIGHUP checks in now, -version prints the build.
- cmd/xtcp2/treatments.go (always) and treatments_embedded.go (tagged);
  checkTreatmentFlags makes the artifact mismatch fatal while runtime
  controller failures stay non-fatal.

Nix and NixOS
- mkGoBinary gains `treatment ? null`: the full destination set (fat
  xtcp2, xtcp2-all) embeds the agent; slim flavors do not.
- xtcp2-treatment-agent joins binaryNames; test-focused-xtcp-treatment
  runs cmd/xtcp2 and the agent binary with the tag.
- services.xtcp2.treatments gains mode (embedded | standalone),
  agentPackage and journalDirectory; standalone adds the
  xtcp2-treatment-agent service and -treatmentAttribution.
- The tcp-controller VM fixture runs standalone with xtcp2-min, so the
  journal watcher is exercised end to end; the Go fixture takes
  -agentUnit for restart/stop/HUP of the agent process.

Tests
- attribution: watcher refresh contracts (journal, missing file, missing
  directory, corrupt, held, location mismatch warns once), rename
  following with removal and re-creation, late directory via poll.
- cmd/xtcp2: checkTreatmentFlags table, attribution mode start/HUP/stop,
  tagged registration test.
- boundaries: the walk honours build tags (go/build MatchFile), so the
  untagged daemon is asserted free of the agent, identity, local and the
  RPC proto, the tagged daemon reaches the agent but never the
  controller, and the standalone binary never reaches the controller.

Docs: agent deployment section and binary table in docs/tcp-controller.md,
build-tag mechanics in docs/build-flavors.md, example standalone unit,
D2 notes and TODOs in docs/status-tcp-controller.md.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
…on has none

The standalone VM run failed its baseline because the slim daemon shipped
location="" and the ClickHouse analysis filters by location. The watcher
now takes the journal's location when -location/LOCATION is unset and keeps
the daemon's own label when set.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Add pkg/treatment/controller/evidence: Source (measure a Window),
CandidateSource (scan a location), Warehouse (both) and the controller's
window policy (WindowFor, SettleDelay, MaxWindow). Maintain and Analyze take
an evidence.Source; Discover takes an evidence.Warehouse and keeps only the
template ASN/prefix scoping, destination cooldown and proposal logic.

Move the ClickHouse client to evidence/clickhouse with analysis.sql and the
former discovery SQL constant as embedded files. Candidates fails on an
undecodable destination instead of skipping it. The controller root no
longer contains SQL, an HTTP client or the ClickHouse name; the boundary
test forbids the adapter from the controller and keeps the port a leaf.

Handler tests use in-memory fakes (fixedEvidence, fakeWarehouse); the
adapter keeps the transport and evidence contract tables, gains a candidate
table, and keeps TestClickHouseAnalysisCounterDeltas for the Nix harness.
Document artifact_sha256 as the digest of the unarchived response.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
…wait for the lock holder in the journal test

The item 5 Nix harness failed because tools/tcp-controller-integration
still named controller.ClickHouse. TestJournalConcurrentExclusion could
also race its temp-dir cleanup against the goroutine still holding the
lock; it now waits for that goroutine before returning.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
…e option

identity.Policy maps full method names to the role a caller must present;
its unary and stream interceptors resolve the peer certificate once, store
the identity in the context, refuse the wrong role, refuse any method
without a declaration (fail closed) and, for agent RPCs, refuse a request
whose agent_id or location is not the certificate's. identity.Agent and
identity.Operator prefer the resolved identity and fall back to the peer,
so handlers and their direct-call tests are unchanged.

controller.RolePolicy derives the table from the (xtcp_treatment.v1.role)
option on both service descriptors; controller.GRPCOptions bundles the
interceptors with the message bounds and main passes it to grpc.NewServer.
TestEveryMethodAgainstEveryRole tries each served method with an agent
certificate, an operator certificate and none; TestRolePolicy fails if a
served method has no declaration. The mTLS bufconn test now runs through
the interceptors.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
…nternal/goip/actuate

Add xtcpnl.RuleWrite, RouteWrite's counterpart for RTM_NEWRULE/RTM_DELRULE:
the fib_rule_hdr from fields, a u32 Table that goes in the header byte up to
255 and in FRA_TABLE above it, and the FRA_* selectors as an ordered list.
The treatment backend's rule() now builds from those fields instead of
spelling `ip rule` keywords and running them through parseRuleWriteArgs;
TestRuleWriteTypedMatchesArgv in internal/goip keeps the two encoders byte
for byte equal for the treatment rule, its v6 and delete forms, table 255,
/32, /128 and `not`.

Move the backend out of the goip root into internal/goip/actuate
(actuate.Backend, RouteTreatment, TCPRouteSettings; actuate.go, rules.go,
routes.go) with its tests and benchmark. rule() refuses out-of-range
table, priority and protocol before encoding. NetlinkSource.ApplyRouteControl
stays in the goip root (route_control.go) because it needs the socket's
unexported descriptor; it remains the only live-write entry point. The goip
hermetic-import test pins the new package in the closure, pkg/treatment/local
imports the actuator, the boundary rule expects it, and the Nix harness runs
./internal/goip/actuate.

Docs: netlink README's "no live kernel writes" callout and the top-level
README sentence now describe the one opted-in actuator path; the controller
doc gains the actuate package row and the typed-rule note; status doc item 7.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
The agent's location is certificate-bound and evidence filters by it, so the
daemon's own -location/LOCATION must agree with it or be absent. Embedded, the
daemon now checks right after loading the agent config and before reading any
credential: an empty daemon location adopts the agent's (logged once); a set
one that differs refuses to start the agent with a message naming both labels,
and socket telemetry continues without it. The record stamp no longer silently
rewrites a configured label, because the only remaining cases are equal or
adopted. Standalone attribution already kept the daemon's label and adopted
the journal's when there was none; unchanged.

main passes c.Location (after envOverrideLabeling) to startTreatments rather
than the raw flag, so LOCATION from the environment counts in both modes.
agent.New names both sides when config and certificate disagree.

Tests: TestCheckTreatmentLocation drives the rule through envOverrideLabeling
(flag, env, equal, differs, env over flag, case); the tagged
TestEmbeddedTreatmentAgentLocation shows unset and equal reach the credential
error while differs is refused before it.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Journal.WithLock took the flock with LOCK_NB, so the daemon's reconcile pass,
the watchdog's expiry pass and a local suspend/resume failed "journal busy"
whenever they met, and the loser skipped its pass; the docs said the lock
serialised them. It now waits: non-blocking attempts with a 1 to 50 ms doubling
delay (a blocking flock cannot be interrupted by a context), returning
ErrJournalBusy wrapping ctx.Err() at the deadline before anything is read or
written. Under the lock it sweeps .pending-* temporaries, which only a writer
that died mid-write can leave. Reconciler.Run passes its ctx, SuspendLocal
takes one, and treatmentctl suspend|resume bounds it at 30 s. status was
already lock-free and stays so.

Tests: TestJournalLockWaits (uncontended, contended-waits, ctx-deadline,
zero-budget), TestJournalLockFreeReadUnderLock, and TestJournalLockHolderDies,
which kills a child test process holding the lock with a half-written temp
file and checks the lock is free, the snapshot intact and the temp swept.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
…ekeeping lease

Agents.DesiredRevisions (a full scan every two seconds) becomes
Agents.DesiredSince(since), reading rows stamped at or after a watermark
through the desired_updated_at index. SaveDesired and Enroll stamp with
clock_timestamp(): now() is transaction start, which precedes the advisory
lock, so it could sort a later commit before an earlier one. The scanner
carries the newest store stamp less a one-second margin; the hub drops
repeats. A step larger than the margin is covered by the agent's next
check-in.

Observe writes one agents row per heartbeat and rewrites the algorithm and
record rows only when a SHA-256 digest of their insert form changes, kept
in the new report_digest column. The previous digest comes back from the
same UPDATE through a FOR UPDATE subquery.

Store.Lease(name, fn) elects which replica runs periodic work: Postgres
takes pg_try_advisory_lock on a dedicated connection and discards the
connection if the unlock fails; memory is a try-lock map. Server.Housekeep
runs Maintain and, when due, Discover under the housekeeping lease and the
controller main loop logs leader transitions.

Tests: storetest rows for DesiredSince (all, changed, equal stamp, none,
store clock) and Lease (free, held elsewhere, released on error,
independent names, cancelled); TestPostgresObserveRewritesOnlyChangedReports
compares row xmin values; TestNotifyWatermark walks a scanner through
change, no change, equal stamp, scan failure, future and behind-watermark
steps; TestHousekeepLease covers the controller side.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
The design document gains a threat-model section: the four principals and
what their credentials allow, defence in depth on the route itself, and the
evidence boundary, where records carry a host-stamped agent ID and location
over the unauthenticated telemetry pipeline. It states what bounds that
today (location and cohort filters, sample minimums, freshness, human
review, immutable evidence, audit) and what is not done (binding records
to the boot ID and collection nonce reported at check-in), which the status
document tracks.

A names section fixes the vocabulary: treatment for the feature, controller
for the server role, route for the kernel object; why xtcp2-treatmentctl is
not folded into xtcp2ctl; the three configuration surfaces and their
audiences. SIGHUP is documented as "reconcile now" only, with the
controller-free treatmentctl pass beside it. The docs hub now says xtcp2
observes by default and names the controller as the one opt-in, reviewed
component that changes host state.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
The lint baseline gate failed on 107 findings in the new controller,
agent and actuator code. Every one is fixed structurally; no nolint
directives, threshold changes or new exclusions.

- Complexity: Reconciler.Run becomes acceptDesired plus a per-pass
  type (assignment, withdraw, tombstones, settle); Maintain becomes
  maintainOne over observe, fetchEvidence, driftPass and trialPass;
  Control splits into controlExperiment and controlHolds; Evaluate
  into windowOutcome, cohortSamples and compare; Backend.Plan into
  destination, rejectLocal, bestMainRoute and supportedAttrs; the
  controller main into flag parsing, store/warehouse setup and a
  housekeep loop; the experiments get into loadCohort/loadPhases.
- Errors: deferred closes, unlocks, rollbacks and Lease's unlock now
  report their failures (Rollback after Commit and a deliberately
  discarded connection are expected and ignored); scan-error paths
  join rows.Close; the journal's atomic write removes its temporary
  file only on failure.
- Contexts: the integration fixture threads the request context;
  the controller listens through net.ListenConfig; pgtest uses
  ExecContext and the test's context.
- Attribution polls sooner (5s) while the journal is unreadable.
- The ClickHouse SQL test finds clickhouse on PATH (the Nix check
  puts it there) instead of an environment-supplied path.
- US spelling, exhaustive switches, shared constants, prealloc.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
List and Get share one loader that reads the experiment rows, cohorts and
phases with id = ANY($1), then every referenced evidence document and its
host rows in two more statements. A 1000-experiment inspect went from about
1 + N x (3 + 2E) round trips to at most six.

pgtest.OpenCounting traces statements through the pgx driver so the bound
is pinned by TestPostgresListStatements without a hook in the store.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
xtcp2-controller, xtcp2-controllerctl, xtcp2-treatment-agent,
xtcp2-treatmentctl and zstd-probe join cmdNames. ipmeta-bootstrap stays
out: its ContinueOnError flag set turns -help into exit 1.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…table

bench-tcp-controller-postgres-indexes runs TestPostgresExperimentQueryIndexes
three times on the shared PostgreSQL fixture and writes the file under
docs/benchmarks; the old file named a deleted test and the pre-rework
index names. The unit target now also writes coverage-table.md from the
go test -cover lines, which replaces the hand-collected three-package table
in docs/tcp-controller-testing.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The row pushed storetest's tx past the comprehensive tier's funlen limit.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Evidence now counts only telemetry records whose (agent_id, host_boot_id,
collection_id) an agent reported over mTLS. The agent owns a per-process
collection nonce: it writes it into the journal header, where a
telemetry-only daemon reads and stamps it, and reports it at check-in as the
now-required Capabilities.collection_id. The watchdog preserves the header's
nonce.

The controller records every session in xtcp_treatment_agent_sessions
(last_seen refreshed at most every five minutes, pruned after eight days by
housekeeping) and passes the population's sessions into every evidence
window. analysis.sql (tcp-host-deltas-v2-bound) measures bound sockets only
and returns unbound_sockets per host, so spoofed or stale records are shown
to the reviewer instead of counted. Discovery proposes only enrolled
candidates.

The VM test gains TCP_CONTROLLER_BINDING_PASS: records copied under a fake
boot and nonce must come back as unbound, and the safety analysis after it
runs with them present.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
An enrolled agent rotating its collection nonce on every check-in could
grow its sessions until the window exceeded evidence.MaxSessions, failing
evidence for the whole cohort. Agents.Sessions now returns at most the
eight most recently seen sessions per agent, so such a host only loses
binding for its own older records. Also records the F5 commit in the
status table.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Binding evidence pushed Discover one over the gocyclo limit; also gofmt on
validate_test.go.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Conflicts: the treatment flags beside main's listener model (S3 TLS
fields, -listener, gRPC listeners); checkTreatmentFlags now runs with
config validation, before anything binds. Service module keeps both the
treatments and listeners options. Tags carry treatmentTags and extraTags.
go.mod takes main's versions plus pgx; vendor hash recomputed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@randomizedcoder
randomizedcoder marked this pull request as ready for review November 14, 2026 05:23
@randomizedcoder
randomizedcoder merged commit 388ab88 into main Nov 14, 2026
1 check passed
randomizedcoder added a commit that referenced this pull request Nov 14, 2026
#282's treatment actuator (internal/goip/actuate) encodes routes and rules
with xtcpnl's typed builders, which FP-8b put behind goip_write.

- BuildWriteRequest, the route builders and the rule builders build with
  `goip_write || treatment`; link, addr, nsid, genl and xfrm writes stay
  goip_write-only.
- actuate, pkg/treatment/{local,agent}, xtcp2-treatment-agent and
  xtcp2-treatmentctl build with `treatment` (mkGoBinary already sets it for
  the full-destination builds those binaries come from).
- Boundary rules for those packages walk with the tag; the untagged daemon
  must not reach actuate.
- golangci tiers lint with `treatment`; tcp-controller quality tests and
  focused-xtcp-treatment pass it (treatmentctl added).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant