Repository navigation
Conversation
Adds the fix-desktop-eval-turn-overhead change: the sut-platynui README gives locators that don't work on a real desktop (checked on Windows 11 with PlatynUI 0.12.0.dev330), keywords_resolve fails suites that wrap library keywords in user keywords, and rf-core hooks add turns the baseline arm never pays. Ticks openrouter-ci-eval-tier tasks 5.2, 5.3, 6.1 and 6.2 with the runs that verified them. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
…verhead - sut-platynui README: @id locators (verified on Windows 11), language-independent window - keywords_resolve: min_calls counts library calls reached through user keywords - new custom.platynui:locators_ok check, gating on the narrow task and in the partial fallback (which now takes a list of checks) - rf-core: formatting drift opt-in (RF_AGENTSKILLS_FORMAT_CHECK); Stop reminder skipped once the suite ran after the last write - rf-platynui: robotcode doc keyword line in Library Import - scripts/desktop-live-check.py and docs/ci/desktop-live-check.md Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
Skill Evaluation: rf-mobile (narrow, treatment, N=3, z-ai/glm-5.3-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 3 · pass: 3 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.02 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Skill Evaluation: rf-desktop (narrow, treatment, N=3, z-ai/glm-5.3-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 3 · pass: 3 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.03 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Skill Evaluation: rf-web (narrow, treatment, N=3, z-ai/glm-5.3-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 6 · pass: 5 · fail: 1 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.03 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Skill Evaluation: rf-api (narrow, treatment, N=3, z-ai/glm-5.3-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 6 · pass: 6 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.04 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Skill Evaluation: rf-core (narrow, treatment, N=3, z-ai/glm-5.3-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 36 · pass: 36 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.25 · turn-limit stops: 3 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Re-baseline after the fixture, grader and hook changes of this PR (N=6, both arms, all tiers, DeepSeek V4.1 Flash, turn factor 1.5). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
z-ai/glm-5.3-flash replaces deepseek/deepseek-v4.1-flash as the only OpenRouter model (output $0.078/M instead of $1.20/M): allow-list, list prices, workflow OPENROUTER_MODEL, tests and docs. Baselines move to baselines/openrouter/glm-5.3-flash/ with the next re-baseline. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the OpenSpec change
fix-desktop-eval-turn-overhead. It explains why rf-desktop's skill arm scores below the baseline arm on the OpenRouter tier, and plans the fix. It also ticks the verifiedopenrouter-ci-eval-tiertasks.This PR holds only the planning artifacts for now. The implementation (tasks 2.3 to 11) lands on this branch next, followed by one re-baseline. It stays a draft until then.
Why
Running the 63 suites from the eval sessions against real Windows Calculator (German Windows 11, PlatynUI 0.12.0.dev330, 2026-11-12) showed:
sut-platynuifixture README gives wrong locators. PlatynUI exposes the UIA AutomationId as@Id;@AutomationIdmatches nothing, and@Nameis localised. All 63 suites failed as written. With the names translated, only 5 suites passed, all from the skill arm. The skill's own example passed both times.min_callsfails 7 correct skill-arm suites that wrapPointer Clickin a user keyword.Planned changes
@Idlocators in the fixture README.platynui_locatorsstatic check.keywords_resolvecounting library calls through user keywords.RF_AGENTSKILLS_FORMAT_CHECK).robotcode doc keywordline moves up inrf-platynui.openrouter-ci-eval-tier
Test plan
openspec validate --strictpasses for both changesscripts/check-docs-voice.pypassestasks.md)🤖 Generated with Claude Code
https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP