Repository navigation
ci(agent-e2e): run on gpt-6-luna; skip live dall-e-3/tts-1 tests - #191
Open
chrishagglund-ship-it wants to merge 7 commits into
Open
chrishagglund-ship-it wants to merge 7 commits into
chrishagglund-ship-it wants to merge 7 commits into
Conversation
The OpenAI key is moving to the Orkes-Developers project, which only allows the gpt-5.6 / gpt-6 families. gpt-4o-mini also shuts down on 2027-01-14. - Set CONDUCTOR_AGENT_LLM_MODEL=openai/gpt-6-luna for every suite. - Suite 11 (LangGraph): build ChatOpenAI from that model, without temperature. Reasoning models reject temperature unless reasoning effort is "none". - Suite 7: skip the live DALL-E 3 and TTS-1 tests. dall-e-3 no longer exists at OpenAI, gpt-image-* is not enabled, and the project gets a 403 for tts-1. - Add an "LLM provider check" first step that calls the model directly, so a dead key or quota fails in seconds, not at the 45-minute timeout. - Add a "Show server errors" step that prints server.log errors to the job log on failure or cancel. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Revert before merge. A server bisect: on 3.34.0-rc2, test_hitl_approve_path stalled twice in the same place, and guardrail matrix #9 timed out once. Run only those two tests, in a matrix against the org-var server version and 3.33.0, the last server the full JS suite passed on (2026-10-26). Co-Authored-By: Claude Opus 5.5 <[email protected]>
Revert before merge. test_hitl_approve_path stays RUNNING after one approval on both 3.34.0-rc2 and 3.33.0 with gpt-6-luna, but passed on 3.33.0 with gpt-4o-mini. After the wait, print every task in its workflow, with full inputs and outputs for the LLM and HUMAN tasks, to see what the model does after the approved publish_article call. Run only that test, still on both servers. Guardrail matrix #9 passed alone on both and goes back to normal. Co-Authored-By: Claude Opus 5.5 <[email protected]>
On gpt-6-luna, test_hitl_approve_path stalled on every run (3.33.0 and 3.34.0-rc2). After the approved publish_article call, the server sends the instructions and the original user request again, after the tool result. Read literally, that is a new request: luna wrote and published a second article, which waits for a second approval the test never gives. gpt-4o-mini treated the request as already done. Keep the task in the user message only. Make the instructions say how to do the work and that one successful publish_article ends the request. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Revert before merge. Make the test's system prompt, tool description
and user message overridable (HITL_SYSTEM, HITL_TOOL_DESC, HITL_USER).
Run one matrix job per variant on the org-var server (3.34.0-rc2), in
place of the 3.33.0 vs rc2 legs. Both servers behaved the same.
- control: current text
- user-stop: stop rule in the user message, which the server
re-sends after the tool results each turn
- tool-once: tool description says it can be called at most once
per chat session
- system-count: system prompt says to count publish_article calls and
end the session when the count is above zero
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Revert before merge. The control prompt failed in 4 of 4 runs. Keep the three passing variants (user-stop, tool-once, system-count) for repeat runs. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Remove the temporary bisect, diagnostic and prompt-variant changes: the workflow and suites go back to their state at 3379c12, and the full suite runs again. On gpt-6-luna, test_hitl_approve_path stalled on both 3.33.0 and 3.34.0-rc2. After the approved publish_article call, the server sends the original user request again, after the tool results. Luna often reads that as a new request and publishes a second article, which waits for a second approval the test never gives. Prompt variants tested in CI (passed / runs): - "publish once; when it returns 'published' you are done": 0/4 - tool description "at most once per chat session": 2/4 - user message "if already published, do not publish again": 3/4 - system prompt: count earlier calls, stop if above zero: 4/4 Use the last one as the agent's instructions. Co-Authored-By: Claude Opus 5.5 <[email protected]>
chrishagglund-ship-it
requested review from
NicholasDCole,
kowser-orkes and
ling-senpeng13
November 13, 2026 16:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The OpenAI key is moving to the Orkes-Developers project, which only allows the gpt-5.6 / gpt-6 families. gpt-4o-mini also shuts down on 2027-01-14.