Repository navigation
fix(scheduler): reject superseded heartbeat receipts on the Python follow-up route - #8708
Conversation
…llow-up route The Python compatibility route for scheduler-ack-current/scheduler-fail-current checked only that the receipt for --turn-instance-id exists; a receipt already superseded by a newer heartbeat Turn still committed scheduler state, while the native TypeScript transaction rejects it as heartbeat_receipt_stale. Thread the Turn from build_scheduler_followup_payload through record_quota_scheduler_ack / record_quota_scheduler_ack_for_decision / record_quota_scheduler_failure_for_decision into _run_native_followup, which now sends turn_instance_id with require_heartbeat_receipt=true. The TypeScript transaction remains the single owner of the missing/stale/superseded fence (it also rechecks under the scheduler state lock); optional parameters keep the in-process API, which has no Turn, unchanged. Fixes loopx-project#8605 Signed-off-by: Lihua <[email protected]>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · reasoning_effort=max
[P2] 过期回执返回未写入前已迁移旧调度状态。 用真实 guard 给同一 Goal 和 Agent 写入先后两个 Turn 回执,只保留合法的旧版路径 scheduler state,再从 Python 入口执行或预览早一轮的 ACK 或 fail。 候选返回 heartbeat_receipt_stale、write_performed=false,旧版文件却已被删除并在哈希路径写出同内容的新文件。File、SQLite 各测执行和预览均复现。直接调用现有 TypeScript 入口时,同组过期请求保留旧文件并且不创建新文件。 开始重建调度决策的位置 先写新路径再删除旧文件
动机 / Motivation
仍走 Python 兼容入口的 scheduler ACK 和失败记录调用方,包括有旧版状态文件的升级环境。 较新的同作用域 Turn 已提交回执。旧 ACK 或失败记录须在写状态之前被拒绝,仍有效的回执可提交或重放。 基线通过 Python 入口可用已被取代的回执提交旧 ACK 或失败记录。重建 hint 时也可能迁移 legacy scheduler state。 候选在 File 和 SQLite 的无旧文件对照中把两种过期调用都返回为 blocked_receipt,现行回执和无 Turn 的兼容调用仍可提交。 证据来自本地真实 Python CLI、TypeScript 子进程和隔离 File/SQLite。测试使用模拟宿主观察值,托管 CI 与真实 App 调度操作未验证。 问题 #8605
改动思路 / Approach
将 Turn 传给已有 TypeScript 回执 owner 是合适的方向。写回事务有前置检查及锁内复查,Python 的重建步骤还需要同一条无副作用边界。 让 Python 重建 hint 时使用现有状态 owner 的只读加载。保留 TypeScript 在验证回执后、锁内进行的迁移和提交。 可以沿用 migrate_legacy=false,让先读 status 再提交的调用也受原回执约束。
具体改动 / Changes
关键代码讲解
build_scheduler_followup_payload 读取原回执并重建 hint,然后把 Turn 传给 ACK 的 record_quota_scheduler_ack 或失败记录函数。_run_native_followup 用原 Turn 启用 TypeScript 的现有检查,原生提交还会在锁内复查。Python 的 hint 重建先经过 loadSchedulerState。旧版文件分支默认迁移,发生在传递 Turn 之前。新增测试从空调度目录运行,覆盖了普通状态写入和有效回执,未进入已有旧版文件的分支。原 Turn 进入原生检查 原生回执检查 锁内复查 新增的公开入口用例
按基线核对旧 Turn 阻断调度写入、当前回执可提交和重放、缺回执与无 Turn 兼容。后两项有本地对照。第一项在只有旧版状态文件的环境仍有缺口。旧回执的写前拒绝与重放契约
对主干的风险 / Risks
四种执行及四种预览对照均确认字段值保留,文件位置已改变。基线也在读取过程中迁移文件,候选接入的原生检查只阻止后续提交,并回报未写入。在同一份回执和原始文件上直接执行 TypeScript 路径,八项都拒绝并保持原文件布局。已有状态的恢复路径也需要遵守新返回结果的边界。
语义与 CI 对齐
源码 Python 检查 80 项通过,TypeScript scheduler 检查 55 项通过,Standard premerge 的 18 项选中检查全部通过。基线与候选各 8 项真实 Python 和原生入口的旧文件对照完成。 另尝试通过生命周期专用的 source-session registry 验证同名 Goal 实例。泛用 quota 入口在基线和候选都先拒绝该 registry,所以该跨实例路径没有到达本次改动。托管 CI 未查询。
我的整体评价 / Verdict
请求修改。保留候选已修好的普通回执检查,让 Python 的前置读取也保持只读,确保在有旧版状态的环境中,过期结果与磁盘状态一致。在旧版文件存在且新路径缺失时,用两个真实 guard 制造旧回执。检查 ACK 和 fail 的 preview、execute 都保留旧文件原字节,当前回执仍可迁移并提交。
English verdict: REQUEST_CHANGES. The Python ACK and failure paths still migrate legacy scheduler state before returning the stale-receipt no-write response, including in preview mode.
…heduler-followup-readonly-8708-20261115 Signed-off-by: LoopX Agent <[email protected]>
Signed-off-by: LoopX Agent <[email protected]>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent / gpt-6.1-sol / OpenAI / runtime_reported / reasoning effort max
动机 / Motivation
复审提交 3f41913,目标主干 8ade78c。涉及仍用 Python 兼容入口记录 Codex App 调度成功或失败的宿主,包括升级后留有旧版调度文件的环境。同一 Goal 与 Agent 已发出两个真实心跳回执,旧回执随后尝试预览或写回。当时只有旧版路径上的调度文件。上一个评审提交能准确拒绝过期回执,但在输出未写状态前就把旧文件迁到新路径。当前主干基线还会接受过期执行。现在 File 和 SQLite 的过期预览及执行均返回 stale,原文件逐字保留且没有创建新文件。当前回执的预览保持只读,实际执行才迁移并提交。本地证据为真实 CLI、TypeScript 事务、隔离 File/SQLite 和受控宿主状态。真实 App 调度与安装态未演练。
改动思路 / Approach
Python 读取调度提示时复用现有只读加载。TypeScript receipt fence 按同一身份在提交前和锁内复查,已有迁移仍由提交事务负责。这个 PR 覆盖 Python 的 ACK、fail 和 App 提示读取,共用原生提交边界。同一分支先重建提示,再用原 Turn 检查并提交。历史状态读取仍保留原值,避免把诊断和预览变成迁移。现行回执可走原写入事务,旧回执的阻断由既有 TypeScript owner 实施。
具体改动 / Changes
验收采用 docs/reference/protocols/host-integration-plugin-plan-v0.md,固定修订 8ade78c。stale-receipt-no-write、hint-read-no-migration、current-and-unbound-commit 均通过。原调度回执协议
关键代码讲解
- build_scheduler_followup_payload 给 ACK 和 fail 都传入选定的 Turn。_run_native_followup 有原 Turn 时请求现有 native receipt fence。实际提交时共享事务在锁内再次核对当前回执。Python follow-up · 原生事务
- load_app_automation_scheduler_state 读取两种路径时都指定 migrate_legacy=False。仍可从旧路径得到原提示。对已经绑定的请求,过期预览、过期执行和当前预览都不增删文件。当前回执执行时才迁移。提示读取 · 原生状态加载
- 当前 ACK 与 fail 的有效回执在 File 和 SQLite 都可提交迁移,无 Turn 的显式兼容执行也可提交。兼容 fail 的第二次执行在三版均返回非零,原 ACK 重放成功。两条重复路径都保持调度文件字节与扣额索引。提交和拒绝回归
对主干的风险 / Risks
独立运行同一公开 CLI 夹具三版,每版 24 个组合。主干基线可接受过期请求,上一评审头拒绝却提前迁移,现头八组过期预览和执行均保留原路径和字节。四组缺回执继续拒绝。所有结果均核对原扣额索引。上一轮请求修改
语义与 CI 对齐
17 项直接 Python 用例、52 项相关 Python 回归、55 项共用 TypeScript 测试通过。Ruff 和 diff 检查通过。PR 复用现有 typed receipt 分类、scope 和 no-spend 边界。兼容 fail 重复调用的原非零返回在三版一致。此次核对使用本地实际 CLI 与隔离存储。
我的整体评价 / Verdict
批准当前提交。原回执拒绝现在与实际磁盘一致,正常当前回执和无 Turn 的兼容写回仍可工作。原回执的前置拒绝和授权时迁移都已在两个后端复验。真实 App 的调度仍待安装态验证。
English verdict: APPROVE. The stale receipt fence now precedes any legacy scheduler state migration, and authorized current follow-ups still commit.
Fixes #8605
Problem
The Python compatibility route for
quota scheduler-ack-current/scheduler-fail-currentimplemented only the existence half of the scheduler ACK receipt contract: with--turn-instance-idit refused only when the receipt was absent (loopx/cli_commands/quota_scheduler_followup.py,find_heartbeat_receipt). A receipt already superseded by a newer heartbeat Turn still rebuilt the decision and committed scheduler state, while the native TypeScript route rejects the same input before any scheduler mutation.Native fence (unchanged, single rule owner):
loopx/control_plane/scheduler/heartbeat_followup.ts:338-348—require_heartbeat_receipt+heartbeatReceiptStatus !== "fresh"→receiptFailure(missing/stale) before host-facts evaluationloopx/control_plane/scheduler/heartbeat_commit.ts:846-861— stale case re-checked under the scheduler state lock at commitloopx/control_plane/rollout_receipt_log.ts:157-198—heartbeatReceiptStatusowns fresh/missing/stale (a later same goal/agent receipt with a differentrun_id⇒ stale)tests/control_plane_ts/scheduler_heartbeat_followup.test.ts:184-255Reproduction (current main
78e936f9d, issue steps)quota should-run --goal-id needs-operator --agent-id codex-side-bypass --codex-app --turn-instance-id TURN1(writes TURN1's receipt)quota_should_runreceipt for TURN2 to the same rollout log ⇒ TURN1 supersededquota scheduler-ack-current ... --turn-instance-id TURN1 --executeBefore (main, reproduced):
The fail route commits the same way once its args are well formed (the exact
failure_hint.cli_argsshape: matching--failed-rrule+--state-key).After (this PR):
Fix — thread the Turn into the native fence (issue shape 1)
No second staleness rule in Python. The Turn now travels from
build_scheduler_followup_payloadthroughrecord_quota_scheduler_ack→record_quota_scheduler_ack_for_decision(andrecord_quota_scheduler_failure_for_decision) into_run_native_followup, which sendsturn_instance_idwithrequire_heartbeat_receipt: trueinstead of hard-codingNone/False. The TypeScript transaction stays the single owner of missing/stale/superseded semantics, including the under-lock recheck; the nativereceiptFailurepayload (quota_scheduler_followup_receipt_failure_v0) flows back through the CLI unchanged, sorc=1and the failure surface come for free. Optional parameters keep the in-process API (no Turn) unchanged; an empty/whitespace Turn id is normalized toNoneat the transport boundary so the request always satisfies the native decode.Python's existing missing pre-check stays: it guards the decision rebuild itself from a receipt-less frontier; the native fence re-owns both cases at commit time.
Tests
New in
tests/control_plane/test_scheduler_ack_current_host_binding.py(red on main, green here):test_scheduler_ack_current_rejects_superseded_turn_receipt_without_state_writetest_scheduler_fail_current_rejects_superseded_turn_receipt_without_state_writetest_scheduler_ack_current_routes_the_newest_turn_receipt(negative twin: a current receipt still commits, state file written)Validation
uv run --extra test python -m pytest tests/control_plane/test_scheduler_ack_current_host_binding.py— 15 passedtest_scheduler_host_failure_cache.py,test_scheduler_backoff_convergence.py,test_scheduler_compat_state_key.py,test_scheduler_fallback_hint.py,test_scheduler_host_followup_hint_transport.py,test_scheduler_ack_decision_table.py— 65 passed;test_heartbeat_receipt.py+test_heartbeat_command_authority.py+test_scheduler_native_launcher.py+test_codex_app_apply_rrule.py+test_superseded_turn_settlement_cli.py— 66 passed, 13 skipped;test_quota_settlement_cli.py+test_turn_envelope.py+test_heartbeat_envelope_transport.py+test_heartbeat_registry_route.py— 290 passednode --test tests/control_plane_ts/scheduler_heartbeat_followup*.test.ts tests/control_plane_ts/scheduler_heartbeat_receipt_fence.test.ts tests/control_plane_ts/scheduler_heartbeat_commit.test.ts— 55 passedruff checkclean on changed filesloopx check --scan-path <4 changed files>: public boundary scan clean (the 8 reported errors are pre-existing personal-registry goal states, identical on pristine main)uv run --extra test loopx canary premerge --from-git-diff --git-diff-base origin/main— passed (18 checks, 0 failures), including RFC bilingual gate and quota-scheduler smokesPre-existing local-only disclosure: all 18 failures of
tests/control_plane/test_scheduler_arbitration_native_cli.pyreproduce identically on pristineorigin/mainin this environment (RegistryProtocolError: rollout event append failedin a should-run fixture, unrelated to the follow-up route); diff of pristine vs patched failure sets is empty.Maintainer follow-up
The Python follow-up could recreate scheduler hints by moving a legacy state file before the native transaction rejected an older Turn. The follow-up now reads both supported App layouts without migrating them. The native commit still validates the receipt and migrates under its existing lock.
Tested commit
3f41913d4d2985d06305bc66da75221986e34c08. Hosted checks and independent review are pending for the updated head. Original finding