Repository navigation
docs(rfc): update benchmark portfolio priority ranking and Bitter-Lesson boundary - #8835
Open
loopx-agent wants to merge 1 commit into
Open
loopx-agent wants to merge 1 commit into
loopx-agent wants to merge 1 commit into
Conversation
…son boundary Add §2.2 summarizing the Bitter-Lesson raw-artifact reset boundary and five failure regimes requiring governed control planes, require a Raw-Artifact Reset Baseline under §5.2 Native baseline for episodic scalar/RSI tracks, add §3.8 with the 2026-11 benchmark portfolio priority matrix (P0/P1/P2/Control-Only) and platform constraints, update §17/§18 in EN/ZH mirrors without touching §6.4, and link the updated ranking from docs/research/README.md. Signed-off-by: LoopX Agent <[email protected]>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Updates the benchmark research program RFC (EN, ZH) and docs/research/README.md with the 2026-11 external benchmark portfolio ranking and Bitter-Lesson boundary synthesis:
arXiv:2603.28052, Verceld0, AlphaEvolve) match or beat lossy hand-written compression, and links./frontier-science-research-program-v0.md(.zh-CN.md) without duplicating §2/§3 content.LHTB 2.0 14/17,RHB arXiv:2605.02964,SWE-Marathon 13.8%), (2) OOD overfitting & prompt/code bloat (RRSI arXiv:2609.24972,arXiv:2605.30621), (3) premature termination & sparse/binary terminal verifiers (zstd-decoder,LHTB covshift), (4) proxy misalignment & tail TLE cliffs (benchmark/proxy_order.py,DuoDiscover arXiv:2610.17184, LangChain Terminal-Bench 2.063.6% -> 53.9%), and (5) stateful dependency-DAG, coverage, and workspace/settlement invariants (LoopsBench,EdgeBench New Foundations,workspace_delivery.py).§5.2item 1 (Native baseline) to require accounting for or running a zero-protocol-tax Raw-Artifact Reset Baseline (all artifacts + scalar scores, clean per-generation reset) on episodic scalar/RSI tracks.EdgeBench,FrontierCS-2,LHTB / LHTB 2.0,LoopsBench), P1 (SWE-Marathon,PaperBenchXpreferred overPaperBench,ALEdistinct fromSakanaAI/ALE-Bench,WideSearch,KernelBench / AlgoTune), P2 (ALE-Bench,StartupBench,MLE-bench,tb-scienceexcluded from RSI due to CI-locked28800sbudget and 100% binaryreward.txt), and Control-Only / Deprioritized (DeepSWE,SWE-bench Pro,CORE-Bench,ScienceAgentBench,BixBench,FrontierScience,RE-Bench) portfolio matrix with task counts, horizons, verifier shapes, frontier headroom, in-repo LOC/readiness, and platform constraints (Linux x86_64 vs macOS ARM64 Rosetta/QEMU/Colima limits, CUDA/GCP dependencies, sharedbenchmark/runtime/+ Stop-hook E0–E2 gates).2026-11-15portfolio/baseline decision row in§17, expands§18with live-verified references, and links§3.8fromdocs/research/README.md.§6.4(Human attention) untouched to stay conflict-free with Capture authenticated LoopX receipts for Research attention #8790.Verification
uv run --extra test python scripts/check_rfc_bilingual.py --base origin/main --head HEAD->RFC bilingual diff check passed.python3 scripts/generate_rfc_status_index.py --check-> exit0git diff --stat origin/main...HEAD-> 3 files changed (+453 / -2), zero changes in§6.4.