Skip to content

Read NATS schedules with dotted identifiers without mixing nested prefixes - #35

Draft
cw-jlu wants to merge 3 commits into
taskiq-python:masterfrom
cw-jlu:fix/read-dotted-schedule-ids
Draft

cw-jlu wants to merge 3 commits into
taskiq-python:masterfrom
cw-jlu:fix/read-dotted-schedule-ids

Conversation

@cw-jlu

@cw-jlu cw-jlu commented Nov 13, 2026 •

Copy link
Copy Markdown

A schedule ID such as daily.report is accepted by ScheduledTask, stored successfully in NATS KV, and then omitted by get_schedules(). Its current prefix.* subject filter matches only one token after the prefix, so a scheduler never sees those jobs in its source snapshot.

Use prefix.> to include multi-token IDs. Check each decoded schedule's ID against its stored KV key so a source with a nested configured prefix does not contribute its schedules to the parent source. The stored key format, schedule ID, serializer, and delete/post-send paths are unchanged. This preserves prefix ownership for valid schedules with distinct stored keys; it does not add an atomic snapshot or a new key encoding scheme.

Fixes #34. Assignment was requested in the issue; no assignee has been set.

Validation actually run

Environment: Windows / Python 3.14.7, repository lock (Taskiq 0.13.0, nats-py 2.12.0), and a real native NATS Server 2.15.3 with JetStream over a private loopback port. The official server archive was checked against its GitHub release SHA-256 digest. Each test service used its own temporary store and was terminated after validation.

  • Final six new regressions copied into an untouched 4c15b00d794d85b3dbb0719fd46ab3028eb1352b worktree: 4 failed, 2 passed. The four failures demonstrate missing dotted IDs for cron/time schedules with both single-token and dotted prefixes. The two passing controls cover nested-prefix isolation and malformed-payload error/recovery.
  • pytest -q -n 2: 23 passed, the complete repository suite on the patch. Real connections, KV keys, and JetStream history requests are used; no NATS/Taskiq component is mocked by the new tests.
  • New tests cover the source API and Taskiq's actual scheduler snapshot helper, exact stored keys, plain/one-dot/multi-dot IDs, delete/re-add with changed arguments, cron retention/time deletion after post_send, and sources sharing a bucket with nested prefixes. Malformed own-prefix payloads still raise and recover after deletion.
  • Whole-repository Black, Ruff, mypy (12 files), all prek run --all-files hooks (including zizmor), and diff whitespace checks passed. Black emits its existing Python 3.15-target warning under Python 3.14 but exits successfully; nats-py emits asyncio.iscoroutinefunction deprecation warnings.

Not run locally: a complete scheduler daemon/worker-process end-to-end test, clustered NATS/failover, older NATS server versions, or the full OS/Python CI matrix. The broader wildcard does not resolve pre-existing key collisions when two prefix/ID combinations encode to the same stored key, and this patch does not change custom bucket history semantics.

Update

A follow-up commit briefly made undecodable multi-token keys be skipped with a warning; it has been reverted. The tree at 49ad0389c97ba928d32e7ec9ef5a598a22d294c5 is identical to 040343c7edd9a6072f28d1c448163d12aa2f2c97, so the validation above still applies, and undecodable payloads under the prefix raise as described.

Known limitation: nested prefixes in one bucket

prefix.> also matches keys written by a source with a nested prefix (for example schedule and schedule.tenant sharing a bucket). The parent decodes those entries with its own serializer before the ownership check. If the nested source uses an incompatible serializer, or stores a malformed payload, the parent's get_schedules() raises. With the previous prefix.* filter, the same configuration worked for single-token IDs.

A key such as schedule.tenant.job cannot be attributed from its name alone: it may be ID tenant.job in schedule, or ID job in schedule.tenant. This patch therefore keeps decoding errors visible instead of skipping them. Sources that share a bucket with nested prefixes need compatible serializers; separate buckets avoid the overlap. I'm happy to switch to skipping and logging such keys if you prefer that trade-off.

A probe against a real NATS Server 2.15.3 (Pickle parent, JSON nested source) shows the parent snapshot raising UnpicklingError on this branch and returning its own schedule on the base.

The follow-up commit and its revert were prepared with Claude (Anthropic) assistance. The PR remains a Draft; no completed human review is claimed.

Review status

AI-assisted investigation, implementation and testing. Please keep this PR in Draft pending independent and account-owner review. No completed human review is claimed.

cw-jlu and others added 3 commits November 13, 2026 14:30
Reading prefix.> also returns keys written by sources with a nested
prefix, which may use another serializer. Skip and log an undecodable
multi-token key instead of failing the whole snapshot; such keys were
never read before multi-token IDs were supported. Undecodable
single-token keys can only belong to this source and still raise.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This reverts commit da0f9c5.

Undecodable payloads under the prefix raise again. Sources sharing a
bucket with nested prefixes need compatible serializers.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NATS schedule source omits stored schedules whose IDs contain dots

1 participant