Add JSON streaming to Function replay watch mode by isaacroldan · Pull Request #9056 · Shopify/cli

isaacroldan · 2026-11-02T13:18:17Z

WHY are these changes introduced?

Function replay watches by default, but rejects --json unless --no-watch is set.

WHAT is this pull request doing?

Add JSONL records for replay status, rebuild results, and native Function results. Send diagnostics through the command event channel on stderr. Preserve native fields, numeric tokens, and strings, and stop replay when stdout closes with EPIPE.

A replay-only file watcher owns scan readiness and startup event completion before JSON replay reports watching. The shared file watcher only exposes a small lifecycle hook; app-event-watcher.ts and the existing text replay hook match main. #9108 moves text replay onto the reusable replay loop and applies the broader watcher startup fixes. This PR can merge independently.

Keep the single native result for --no-watch --json. Document the stream contract and expose FunctionReplayRecord in --json-schema.

How to manually test your changes?

Use a test app with a Function and a saved Function log.

Run pnpm shopify app function replay --path <function-directory> --log <log-id> --json --no-input > replay.jsonl 2> replay.stderr.
Edit the Function, introduce a build error, then fix it. Check the failed build record's error on stdout and one failure diagnostic on stderr. Replay should keep watching after rebuild and completed Function failures.
Press Ctrl-C while idle. Check the final cancelled record and exit code 130.
Run the same JSON replay command with | head -n 1. Check that the pipeline finishes.
Repeat without --json to check the existing text workflow. Repeat with --no-watch --json to check the single native runner result.
Run pnpm shopify app function replay --json-schema. Check the record types and the distinction between Function run and process exit codes without starting replay.

Validation

All 28 current GitHub checks passed on 52e5840f26: Linux/macOS/Windows Node matrices, GraphQL freshness, type checking, lint, bundle, unused-code checks, generated help, and both E2E shards. CI run. The Nx build crash in the first E2E shard passed on retry.
After rebasing onto current main, all 50 focused replay and command tests passed. Both local fixture TOML tests passed. Uncached schema generation completed with no tracked changes.
Real watcher tests cover immediate edits, startup rescans, native errors, cancellation before watcher creation, and JSON command context. The consolidated JSON tests run the real loop and output writer together; string and startup completion mutations fail. The existing text hook, component, and hook tests match main exactly.
App type checking, CLI build, scoped ESLint, formatting, and diff checks passed. Current main already includes the E2E app proxy scope and generated organization scalar fixes. Manifest and README regeneration made no additional changes.
Local checks ran on macOS ARM with Node 24.22.1. Native replay probes use a controlled runner capture. Hosted CI covers the other supported systems and Node versions.

Checklist

I've considered possible cross-platform impacts (Mac, Linux, Windows)
I've considered possible documentation changes
I've considered analytics changes to measure impact
The change is user-facing and has changesets.

isaacroldan · 2026-11-03T15:17:47Z

/snapit

github-actions · 2026-11-03T15:21:09Z

🫰✨ Thanks @isaacroldan! Your snapshot has been published to npm.

Built from ed6ab2e57bd53a2313b61b110a8af48d9ee8e33d. Workflow run.

Test the snapshot by installing your package globally:

pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20261103151843

Caution

After installing, validate the version by running shopify version in your terminal.
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

isaacroldan · 2026-11-04T12:52:31Z

/snapit

github-actions · 2026-11-04T12:56:04Z

🫰✨ Thanks @isaacroldan! Your snapshot has been published to npm.

Built from fe3d5e5cd486a829c4b16c480012b6ecc051f1b1. Workflow run.

Test the snapshot by installing your package globally:

pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20261104125335

Caution

After installing, validate the version by running shopify version in your terminal.
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

dmerand

This change looks great overall! Very exciting to get this in. I've got some agent comments that are worth considering before we merge. Nits can be addressed at your discretion.

dmerand · 2026-11-06T18:54:20Z

+    process.exitCode = {SIGINT: 130, SIGTERM: 143, SIGQUIT: 131}[receivedSignal]
+    options.abortController.abort()
+  }
+  process.on('shopify:signal', onSignal)


Replay does not stop when stdout closes. Oclif catches the stdout EPIPE error, and this writer does not abort the loop. So shopify app function replay --json ... | head -1 leaves the watcher running, and a shell pipeline waits for it. More edits still start builds.

I reproduced this with the real watch loop and file watcher. Only the runner was mocked. The process stopped only after SIGINT. #9057 does not change this file.

Can you abort replay on a stdout EPIPE error, and remove that listener in finally?

Fixed: stdout EPIPE aborts replay, suppresses later stdout records, and removes the error listener in finally. The regression test uses the real replay loop and output writer. A separate test with a real OS pipe and native file watcher ended with exit code 0, with no new builds after cancellation and no added error listener left behind. Active runner and build cancellation remains in #9057, which is rebased onto this fix.

The review fixes are in 86775f2b5a and 585c0f3a5d.

🤖 Generated by the review-pr skill

dmerand · 2026-11-06T18:54:20Z

+
+Watches for changes by default. Use --no-watch to run once.
+
+Use --json for JSONL output while watching, or --no-watch --json for a single JSON result.`


Please add the replay stream to the streaming section of docs/cli/json-output.md. That section has a paragraph for app dev, and main also has one for theme dev.

Describe the three record types and the stream that gets each one. Also describe the lifecycle (watching, cancelled), recoverable and fatal failures, and the exit codes 130, 143, and 131. This help text says JSONL, but it does not give this contract.

Added the replay contract to the streaming section of docs/cli/json-output.md. It describes the three stdout record types, stderr diagnostics, watching and cancelled, recoverable and fatal failures, exit codes 130/143/131, pipe closure, and finite --no-watch --json output. The PR description also explains these changes.

The review fixes are in 86775f2b5a and 585c0f3a5d.

🤖 Generated by the review-pr skill

dmerand · 2026-11-06T18:54:20Z

+  .object({
+    type: zod.literal('execution'),
+    timestamp: jsonOutputTimestampSchema,
+    result: functionRunJsonOutputSchema.schema,


The execution record reuses FunctionRunResult. Its success description says "the command exits nonzero" (runner/types.ts:27). In watch mode this is not true, because replay continues to watch.

Please override the description here, or make the shared text true for both modes. Please also describe exitCode on the next line. It is the exit code of the Function run, not of the process.

Updated the shared success description to describe a completed Function execution failure without claiming that watch replay exits. The execution record now describes exitCode as the Function run's exit code. Regenerated the README and manifest, and checked the descriptions through the full command's --json-schema output.

The review fixes are in 86775f2b5a and 585c0f3a5d.

🤖 Generated by the review-pr skill

dmerand · 2026-11-06T18:54:20Z

+    await watcher.start({stdout: diagnostics, stderr: diagnostics, signal}, false)
+    if (!signal.aborted) {
+      onEvent({type: 'status', status: 'watching'})
+      await enqueue(extension)


Nit (non-blocking): watcher.onEvent is registered before watcher.start(). So a rebuild during the initial scan can run before watching. Then this line runs the original extension, not the rebuilt one.

A probe that holds Chokidar's ready event gave build, execution, watching, execution. I could not reproduce it without the hold, so the window is small. Can startup rebuilds wait for watching, and can this line use the latest extension?

Startup rebuild records and executions wait for watching, and the first execution uses the latest rebuilt extension. A failed startup rebuild waits for recovery. Tests cover successful and failed startup rebuilds.

After the scope split, #9056 opts in to readiness and startup event completion only for JSON replay. The broader watcher defaults, startup build result handling, and text replay loop reuse are in #9108. #9056 can merge independently.

🤖 Generated by the review-pr skill

dmerand · 2026-11-06T18:54:20Z

+      JSON.stringify(functionReplayEventSchema.validate({type: event.type, timestamp, status: event.status})),
+    )
+    if (event.type === 'build' && event.status === 'failed') {
+      renderCommandEventAsJson({type: 'diagnostic', timestamp, level: 'error', message: event.error})


Nit (non-blocking): A build failure writes the same message two times on stderr. AppEventWatcher writes it to the diagnostics stream (level: "warning", line 26), and this line writes it again (level: "error"). A real CLI run showed both lines.

Question: can the failed build record carry the message instead? Then a consumer that reads only stdout can see why the build failed, and stderr has one copy.

The failed stdout build record now includes its message in error. Removed the second stderr emission from that record. A test with the real replay loop and native file watcher confirmed one failure diagnostic on stderr and the failure message in the stdout build record.

The review fixes are in 86775f2b5a and 585c0f3a5d.

🤖 Generated by the review-pr skill

dmerand · 2026-11-06T18:54:20Z

+      }
+      watcher.once('ready', onReady)
+      watcher.once('error', onError)
+      this.options.signal.addEventListener('abort', onReady, {once: true})


Question (non-blocking): start() now waits for the initial scan, and an abort resolves the wait here. After that, AppEventWatcher.start() still emits ready. So in app dev, Ctrl-C during the scan can still let DevSession.onStart bundle and upload. Before this PR, this window was very short.

#9057 adds an abort check after fileWatcher.start() in app-event-watcher.ts. Will both PRs ship in the same release? If not, can this PR take that one-line guard?

The PR description and the changeset mention only replay. Please also mention this app dev startup change.

After the scope split, #9056 leaves the default watcher and app dev startup behavior unchanged. Only JSON replay opts in to scan readiness and startup event completion, with cancellation checked before it reports watching.

The general app dev abort guard and latest startup build result handling are now in #9108, with their patch changesets and regression tests. #9056 does not depend on that follow-up for JSON support. #9057 follows #9108 and contains active process/build shutdown.

🤖 Generated by the review-pr skill

dmerand · 2026-11-06T18:54:20Z

+  onTestFinished(() => {
+    watcherCallback.emitDestroy()
+  })
+  const resultJson = `{


Nit (non-blocking): The regex in json.ts:37-40 rewrites native runner output, but this fixture has no space and no escaped quote inside a string. If the regex strips spaces inside strings, or ignores \", all tests still pass. I checked both changes locally.

Please add a value such as "logs": "hello world \"quoted\" done", and compare the parsed result with the parsed input.

Expanded the fixture to hello world \"quoted words\" done with a newline, and compare the parsed execution result with the parsed native input. The space inside the escaped quotes is needed to catch a regex that treats those quotes as string boundaries. Both faulty mutations now fail: stripping spaces inside strings and ignoring escaped quotes. The separate assertion still checks that the unsafe integer token is preserved.

The review fixes are in 86775f2b5a and 585c0f3a5d.

🤖 Generated by the review-pr skill

dmerand · 2026-11-06T18:54:20Z

+  .strict()
+
+export const functionReplayEventSchema = defineJsonOutputSchema({
+  name: 'FunctionReplayEvent',


Nit (non-blocking): In --json-schema, Event means stderr side events. The stdout records here are named FunctionReplayEvent. The GraphiQL stream names its stdout records GraphiQLRecord.

Can this be FunctionReplayRecord? Replay --json is not released yet, so the rename costs little now.

Renamed the stdout schema and its definition to FunctionReplayRecord. Updated the writer, schema tests, documentation, generated README, and manifest. The full command's --json-schema output confirms the new name; Event continues to describe stderr side events.

The review fixes are in 86775f2b5a and 585c0f3a5d.

🤖 Generated by the review-pr skill

dmerand · 2026-11-06T19:03:49Z

+  if (event.type === 'diagnostic') {
+    const message = unstyled(event.message)
+    if (message.trim().length > 0) {
+      renderCommandEventAsJson({type: 'diagnostic', timestamp, level: 'warning', message})


Question (non-blocking): Did you consider sending these diagnostics through the command event channel (emitCommandEvent)? Lines 26 and 49 call renderCommandEventAsJson directly. Other streams, for example the GraphiQL session (graphiql/session.ts:115), use the channel.

The JSON output is the same today. With the channel, the timestamp comes from the channel clock, and a test sink or a future sink also gets these events.

Diagnostics now use emitCommandEvent. Bound the replay output callback to its command context so native watcher callbacks retain the channel. A test invokes the callback from a separate async resource and checks that the custom sink receives the diagnostics and that stderr uses the channel clock's timestamp.

The review fixes are in 86775f2b5a and 585c0f3a5d.

🤖 Generated by the review-pr skill

isaacroldan · 2026-11-09T16:09:59Z

/snapit

github-actions · 2026-11-09T16:13:28Z

🫰✨ Thanks @isaacroldan! Your snapshot has been published to npm.

Built from 7faa5bb1f4a14c58cb2df14c7447da64293a25d0. Workflow run.

Test the snapshot by installing your package globally:

pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20261109161059

Caution

After installing, validate the version by running shopify version in your terminal.
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

github-actions Bot added the Area: @shopify/cli @shopify/cli package issues label Nov 2, 2026

isaacroldan added this pull request to stack #9058 November 2, 2026 13:19

isaacroldan marked this pull request as ready for review November 3, 2026 12:38

isaacroldan requested review from a team as code owners November 3, 2026 12:38

isaacroldan force-pushed the codex/function-replay-json branch from 61c9a3a to 6f38ba4 Compare November 3, 2026 13:13

isaacroldan force-pushed the codex/function-replay-json branch from ed6ab2e to fe3d5e5 Compare November 4, 2026 12:51

isaacroldan mentioned this pull request Nov 5, 2026

Add JSON streaming to app function replay watch mode #9007

Closed

4 tasks done

dmerand reviewed Nov 6, 2026

View reviewed changes

isaacroldan mentioned this pull request Nov 9, 2026

Share replay watch logic and fix watcher startup #9108

Draft

1 of 4 tasks

isaacroldan removed this pull request from stack #9058 November 9, 2026 12:49

isaacroldan added this pull request to stack #9109 November 9, 2026 12:50

isaacroldan mentioned this pull request Nov 9, 2026

Fix Function replay shutdown and process cancellation #9057

Draft

1 of 4 tasks

dmerand approved these changes Nov 9, 2026

View reviewed changes

isaacroldan added 10 commits November 9, 2026 18:05

Add JSON streaming to Function replay watch mode

da0935b

Share Function replay watch logic across text and JSON output

9cf6ce4

Remove unused replay type and normalize generated manifest

ad427fe

Simplify Function replay watch tests

41811b0

Keep Function replay diagnostics in JSON format

a53464f

Fix Function replay startup and error handling

4ac13f6

Capture watcher readiness once in restart tests

101e6a1

Address Function replay streaming review feedback

975c4e8

Preserve startup rebuild results before watcher readiness

8000cd7

Limit replay JSON changes to opt-in watcher readiness

ad0a39e

Keep watcher readiness and coordination local to replay

52e5840

isaacroldan removed this pull request from stack #9109 November 9, 2026 17:12

isaacroldan force-pushed the codex/function-replay-json branch from 7faa5bb to 52e5840 Compare November 9, 2026 17:13

isaacroldan added this pull request to stack #9110 November 9, 2026 17:13


		Watches for changes by default. Use --no-watch to run once.

		Use --json for JSONL output while watching, or --no-watch --json for a single JSON result.`

Conversation

isaacroldan commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

WHY are these changes introduced?

WHAT is this pull request doing?

How to manually test your changes?

Validation

Checklist

Uh oh!

isaacroldan commented Nov 3, 2026

Uh oh!

github-actions Bot commented Nov 3, 2026

Uh oh!

isaacroldan commented Nov 4, 2026

Uh oh!

github-actions Bot commented Nov 4, 2026

Uh oh!

dmerand left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isaacroldan Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isaacroldan Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isaacroldan commented Nov 9, 2026

Uh oh!

github-actions Bot commented Nov 9, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

isaacroldan commented Nov 2, 2026 •

edited

Loading

isaacroldan Nov 9, 2026 •

edited

Loading

isaacroldan Nov 9, 2026 •

edited

Loading