Skip to content

fix(scheduler): reject superseded heartbeat receipts on the Python follow-up route - #8708

Merged
loopx-agent merged 3 commits into
loopx-project:mainfrom
LIHUA919:codex/scheduler-superseded-receipt-8605
Nov 15, 2026
Merged

loopx-agent merged 3 commits into
loopx-project:mainfrom
LIHUA919:codex/scheduler-superseded-receipt-8605

Conversation

@LIHUA919

@LIHUA919 LIHUA919 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #8605

Problem

The Python compatibility route for quota scheduler-ack-current / scheduler-fail-current implemented only the existence half of the scheduler ACK receipt contract: with --turn-instance-id it refused only when the receipt was absent (loopx/cli_commands/quota_scheduler_followup.py, find_heartbeat_receipt). A receipt already superseded by a newer heartbeat Turn still rebuilt the decision and committed scheduler state, while the native TypeScript route rejects the same input before any scheduler mutation.

Native fence (unchanged, single rule owner):

  • loopx/control_plane/scheduler/heartbeat_followup.ts:338-348 — require_heartbeat_receipt + heartbeatReceiptStatus !== "fresh" → receiptFailure (missing/stale) before host-facts evaluation
  • loopx/control_plane/scheduler/heartbeat_commit.ts:846-861 — stale case re-checked under the scheduler state lock at commit
  • loopx/control_plane/rollout_receipt_log.ts:157-198 — heartbeatReceiptStatus owns fresh/missing/stale (a later same goal/agent receipt with a different run_id ⇒ stale)
  • pinned by tests/control_plane_ts/scheduler_heartbeat_followup.test.ts:184-255

Reproduction (current main 78e936f9d, issue steps)

  1. quota should-run --goal-id needs-operator --agent-id codex-side-bypass --codex-app --turn-instance-id TURN1 (writes TURN1's receipt)
  2. append a later quota_should_run receipt for TURN2 to the same rollout log ⇒ TURN1 superseded
  3. quota scheduler-ack-current ... --turn-instance-id TURN1 --execute

Before (main, reproduced):

python find_heartbeat_receipt(TURN1): FOUND
ts heartbeatReceiptStatus(TURN1): {"status":"stale"}
ack TURN1 rc: 0
ack TURN1 ok: True
ack TURN1 scheduler_state_mutated: True
ack TURN1 reason: updated scheduler state ack: needs-operator/codex-side-bypass applied FREQ=MINUTELY;INTERVAL=30
control (unknown turn) rc: 1 status: heartbeat_receipt_missing state: blocked_receipt

The fail route commits the same way once its args are well formed (the exact failure_hint.cli_args shape: matching --failed-rrule + --state-key).

After (this PR):

ack TURN1 rc: 1
status: heartbeat_receipt_stale   state: blocked_receipt
error_code: SCHEDULER_FOLLOWUP_HEARTBEAT_RECEIPT_STALE
scheduler_state_mutated: False    (no scheduler-state file written)

Fix — thread the Turn into the native fence (issue shape 1)

No second staleness rule in Python. The Turn now travels from build_scheduler_followup_payload through record_quota_scheduler_ack → record_quota_scheduler_ack_for_decision (and record_quota_scheduler_failure_for_decision) into _run_native_followup, which sends turn_instance_id with require_heartbeat_receipt: true instead of hard-coding None/False. The TypeScript transaction stays the single owner of missing/stale/superseded semantics, including the under-lock recheck; the native receiptFailure payload (quota_scheduler_followup_receipt_failure_v0) flows back through the CLI unchanged, so rc=1 and the failure surface come for free. Optional parameters keep the in-process API (no Turn) unchanged; an empty/whitespace Turn id is normalized to None at the transport boundary so the request always satisfies the native decode.

Python's existing missing pre-check stays: it guards the decision rebuild itself from a receipt-less frontier; the native fence re-owns both cases at commit time.

Tests

New in tests/control_plane/test_scheduler_ack_current_host_binding.py (red on main, green here):

  • test_scheduler_ack_current_rejects_superseded_turn_receipt_without_state_write
  • test_scheduler_fail_current_rejects_superseded_turn_receipt_without_state_write
  • test_scheduler_ack_current_routes_the_newest_turn_receipt (negative twin: a current receipt still commits, state file written)

Validation

  • uv run --extra test python -m pytest tests/control_plane/test_scheduler_ack_current_host_binding.py — 15 passed
  • related scheduler ACK/heartbeat suites: test_scheduler_host_failure_cache.py, test_scheduler_backoff_convergence.py, test_scheduler_compat_state_key.py, test_scheduler_fallback_hint.py, test_scheduler_host_followup_hint_transport.py, test_scheduler_ack_decision_table.py — 65 passed; test_heartbeat_receipt.py + test_heartbeat_command_authority.py + test_scheduler_native_launcher.py + test_codex_app_apply_rrule.py + test_superseded_turn_settlement_cli.py — 66 passed, 13 skipped; test_quota_settlement_cli.py + test_turn_envelope.py + test_heartbeat_envelope_transport.py + test_heartbeat_registry_route.py — 290 passed
  • native contract unchanged: node --test tests/control_plane_ts/scheduler_heartbeat_followup*.test.ts tests/control_plane_ts/scheduler_heartbeat_receipt_fence.test.ts tests/control_plane_ts/scheduler_heartbeat_commit.test.ts — 55 passed
  • ruff check clean on changed files
  • loopx check --scan-path <4 changed files>: public boundary scan clean (the 8 reported errors are pre-existing personal-registry goal states, identical on pristine main)
  • uv run --extra test loopx canary premerge --from-git-diff --git-diff-base origin/main — passed (18 checks, 0 failures), including RFC bilingual gate and quota-scheduler smokes
  • No TS changes, so no TS typecheck surface changed

Pre-existing local-only disclosure: all 18 failures of tests/control_plane/test_scheduler_arbitration_native_cli.py reproduce identically on pristine origin/main in this environment (RegistryProtocolError: rollout event append failed in a should-run fixture, unrelated to the follow-up route); diff of pristine vs patched failure sets is empty.

Maintainer follow-up

The Python follow-up could recreate scheduler hints by moving a legacy state file before the native transaction rejected an older Turn. The follow-up now reads both supported App layouts without migrating them. The native commit still validates the receipt and migrates under its existing lock.

Evidence Result
File and SQLite source CLI, ACK and failure commands On the previous head, 8 stale preview and execution cases moved the legacy file despite reporting no write. On this head all 8 reject and preserve the original bytes and path. The 4 current-receipt execution cases still migrate and commit. The tracked regression also checks a current preview.
Scope Preserves the original contributor commit, receipt scope and native fence. The separate GoalRef work in #8606 remains independently reviewable.
Local validation 296 Python tests, 70 native scheduler tests, Ruff and mypy, and 18 selected premerge checks passed.

Tested commit 3f41913d4d2985d06305bc66da75221986e34c08. Hosted checks and independent review are pending for the updated head. Original finding

…llow-up route

The Python compatibility route for scheduler-ack-current/scheduler-fail-current
checked only that the receipt for --turn-instance-id exists; a receipt already
superseded by a newer heartbeat Turn still committed scheduler state, while the
native TypeScript transaction rejects it as heartbeat_receipt_stale.

Thread the Turn from build_scheduler_followup_payload through
record_quota_scheduler_ack / record_quota_scheduler_ack_for_decision /
record_quota_scheduler_failure_for_decision into _run_native_followup, which now
sends turn_instance_id with require_heartbeat_receipt=true. The TypeScript
transaction remains the single owner of the missing/stale/superseded fence (it
also rechecks under the scheduler state lock); optional parameters keep the
in-process API, which has no Turn, unchanged.

Fixes loopx-project#8605

Signed-off-by: Lihua <[email protected]>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · reasoning_effort=max

评审提交 84377d7,目标基线为 78e936f。

[P2] 过期回执返回未写入前已迁移旧调度状态。 用真实 guard 给同一 Goal 和 Agent 写入先后两个 Turn 回执,只保留合法的旧版路径 scheduler state,再从 Python 入口执行或预览早一轮的 ACK 或 fail。 候选返回 heartbeat_receipt_stale、write_performed=false,旧版文件却已被删除并在哈希路径写出同内容的新文件。File、SQLite 各测执行和预览均复现。直接调用现有 TypeScript 入口时,同组过期请求保留旧文件并且不创建新文件。 开始重建调度决策的位置 先写新路径再删除旧文件

动机 / Motivation

仍走 Python 兼容入口的 scheduler ACK 和失败记录调用方,包括有旧版状态文件的升级环境。 较新的同作用域 Turn 已提交回执。旧 ACK 或失败记录须在写状态之前被拒绝,仍有效的回执可提交或重放。 基线通过 Python 入口可用已被取代的回执提交旧 ACK 或失败记录。重建 hint 时也可能迁移 legacy scheduler state。 候选在 File 和 SQLite 的无旧文件对照中把两种过期调用都返回为 blocked_receipt,现行回执和无 Turn 的兼容调用仍可提交。 证据来自本地真实 Python CLI、TypeScript 子进程和隔离 File/SQLite。测试使用模拟宿主观察值,托管 CI 与真实 App 调度操作未验证。 问题 #8605

改动思路 / Approach

将 Turn 传给已有 TypeScript 回执 owner 是合适的方向。写回事务有前置检查及锁内复查,Python 的重建步骤还需要同一条无副作用边界。 让 Python 重建 hint 时使用现有状态 owner 的只读加载。保留 TypeScript 在验证回执后、锁内进行的迁移和提交。 可以沿用 migrate_legacy=false,让先读 status 再提交的调用也受原回执约束。

具体改动 / Changes

关键代码讲解

build_scheduler_followup_payload 读取原回执并重建 hint,然后把 Turn 传给 ACK 的 record_quota_scheduler_ack 或失败记录函数。_run_native_followup 用原 Turn 启用 TypeScript 的现有检查,原生提交还会在锁内复查。Python 的 hint 重建先经过 loadSchedulerState。旧版文件分支默认迁移,发生在传递 Turn 之前。新增测试从空调度目录运行,覆盖了普通状态写入和有效回执,未进入已有旧版文件的分支。原 Turn 进入原生检查 原生回执检查 锁内复查 新增的公开入口用例

按基线核对旧 Turn 阻断调度写入、当前回执可提交和重放、缺回执与无 Turn 兼容。后两项有本地对照。第一项在只有旧版状态文件的环境仍有缺口。旧回执的写前拒绝与重放契约

对主干的风险 / Risks

四种执行及四种预览对照均确认字段值保留,文件位置已改变。基线也在读取过程中迁移文件,候选接入的原生检查只阻止后续提交,并回报未写入。在同一份回执和原始文件上直接执行 TypeScript 路径,八项都拒绝并保持原文件布局。已有状态的恢复路径也需要遵守新返回结果的边界。

语义与 CI 对齐

源码 Python 检查 80 项通过,TypeScript scheduler 检查 55 项通过,Standard premerge 的 18 项选中检查全部通过。基线与候选各 8 项真实 Python 和原生入口的旧文件对照完成。 另尝试通过生命周期专用的 source-session registry 验证同名 Goal 实例。泛用 quota 入口在基线和候选都先拒绝该 registry,所以该跨实例路径没有到达本次改动。托管 CI 未查询。

我的整体评价 / Verdict

请求修改。保留候选已修好的普通回执检查,让 Python 的前置读取也保持只读,确保在有旧版状态的环境中,过期结果与磁盘状态一致。在旧版文件存在且新路径缺失时,用两个真实 guard 制造旧回执。检查 ACK 和 fail 的 preview、execute 都保留旧文件原字节,当前回执仍可迁移并提交。

English verdict: REQUEST_CHANGES. The Python ACK and failure paths still migrate legacy scheduler state before returning the stale-receipt no-write response, including in preview mode.

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent / gpt-6.1-sol / OpenAI / runtime_reported / reasoning effort max

动机 / Motivation

复审提交 3f41913,目标主干 8ade78c。涉及仍用 Python 兼容入口记录 Codex App 调度成功或失败的宿主,包括升级后留有旧版调度文件的环境。同一 Goal 与 Agent 已发出两个真实心跳回执,旧回执随后尝试预览或写回。当时只有旧版路径上的调度文件。上一个评审提交能准确拒绝过期回执,但在输出未写状态前就把旧文件迁到新路径。当前主干基线还会接受过期执行。现在 File 和 SQLite 的过期预览及执行均返回 stale,原文件逐字保留且没有创建新文件。当前回执的预览保持只读,实际执行才迁移并提交。本地证据为真实 CLI、TypeScript 事务、隔离 File/SQLite 和受控宿主状态。真实 App 调度与安装态未演练。

改动思路 / Approach

Python 读取调度提示时复用现有只读加载。TypeScript receipt fence 按同一身份在提交前和锁内复查,已有迁移仍由提交事务负责。这个 PR 覆盖 Python 的 ACK、fail 和 App 提示读取,共用原生提交边界。同一分支先重建提示,再用原 Turn 检查并提交。历史状态读取仍保留原值,避免把诊断和预览变成迁移。现行回执可走原写入事务,旧回执的阻断由既有 TypeScript owner 实施。

具体改动 / Changes

验收采用 docs/reference/protocols/host-integration-plugin-plan-v0.md,固定修订 8ade78c。stale-receipt-no-write、hint-read-no-migration、current-and-unbound-commit 均通过。原调度回执协议

关键代码讲解

  • build_scheduler_followup_payload 给 ACK 和 fail 都传入选定的 Turn。_run_native_followup 有原 Turn 时请求现有 native receipt fence。实际提交时共享事务在锁内再次核对当前回执。Python follow-up · 原生事务
  • load_app_automation_scheduler_state 读取两种路径时都指定 migrate_legacy=False。仍可从旧路径得到原提示。对已经绑定的请求,过期预览、过期执行和当前预览都不增删文件。当前回执执行时才迁移。提示读取 · 原生状态加载
  • 当前 ACK 与 fail 的有效回执在 File 和 SQLite 都可提交迁移,无 Turn 的显式兼容执行也可提交。兼容 fail 的第二次执行在三版均返回非零,原 ACK 重放成功。两条重复路径都保持调度文件字节与扣额索引。提交和拒绝回归

对主干的风险 / Risks

独立运行同一公开 CLI 夹具三版,每版 24 个组合。主干基线可接受过期请求,上一评审头拒绝却提前迁移,现头八组过期预览和执行均保留原路径和字节。四组缺回执继续拒绝。所有结果均核对原扣额索引。上一轮请求修改

语义与 CI 对齐

17 项直接 Python 用例、52 项相关 Python 回归、55 项共用 TypeScript 测试通过。Ruff 和 diff 检查通过。PR 复用现有 typed receipt 分类、scope 和 no-spend 边界。兼容 fail 重复调用的原非零返回在三版一致。此次核对使用本地实际 CLI 与隔离存储。

我的整体评价 / Verdict

批准当前提交。原回执拒绝现在与实际磁盘一致,正常当前回执和无 Turn 的兼容写回仍可工作。原回执的前置拒绝和授权时迁移都已在两个后端复验。真实 App 的调度仍待安装态验证。

English verdict: APPROVE. The stale receipt fence now precedes any legacy scheduler state migration, and authorized current follow-ups still commit.

@loopx-agent
loopx-agent merged commit e8cb0a8 into loopx-project:main Nov 15, 2026
34 of 50 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: the Python scheduler follow-up route accepts a heartbeat receipt a newer Turn superseded

2 participants