Autoresearch: the family, and the evidence behind it
Short answer: a propose, evaluate, refine loop is worth running only where a grader exists, and what keeps it honest is selection discipline plus a process record.
An autoresearch agent runs a repeated cycle: propose a candidate, evaluate it against something checkable, keep or discard, and propose again, for hours or longer, with the score deciding what survives. Two things make such a loop trustworthy rather than merely plausible. The first is how the winner is chosen, because choosing on the same measurements you iterated against inflates the result. The second is whether anyone can inspect how the work was done, because a finished write-up hides the intermediate decisions that carry the faults.
The loop contract
Five rules. Breaking any one of them turns a loop into a story that happens to have numbers in it.
- Fix the artefact and the grader before the first candidate exists. Name the file, the schema, the scoring script or the fixture, and the command that scores it. If the grader cannot be written down in one line, there is not one yet.
- One episode is bounded. Propose, evaluate, refine, stop. Interventions, comparisons and deletions belong at episode boundaries, never mid-episode. An unbounded episode cannot be scored, so it cannot be selected between.
- Every episode leaves a ledger line: episode id, what changed, the score, the grader command or hash, and the decision. The ledger is the run, and it is also the only thing an independent audit can read [11].
- Selection is separate from production. The producer does not certify its own candidate. Either a second party re-grades, or the score is reported as self-assessed and labelled that way.
- Report the delta with its denominator. A fixture score of 24 of 30, previously 21 of 30, is a result. The word improved is not.
Three ways a loop fails, and the discipline for each
Each is documented rather than hypothetical. The middle column is what an operator actually sees when it happens.
| Failure | What it looks like | Discipline |
|---|---|---|
| Idea basin | Later episodes refine the same approach. Independent runs on the same task plateau at markedly different scores, and the gaps survive extra compute [1]. | Fork into fresh-context branches from the current workspace, keep the best, repeat on a fixed schedule at budget fractions [1]. Fixed cadence beats plateau detection, because stall clocks mostly never fire. |
| Selection noise (the winner's curse) | The best candidate wins largely because the evaluation set was small and noisy. The score does not survive on new items [4]. | Split the fixture before iterating: rows used to choose, and rows read only to report. Report both numbers separately, with denominators. |
| Uncertified verifier | The loop starts passing because the grader got easier, not because the work got better. A collapsed, always-pass verifier still trains the task successfully, so the task score cannot detect it [6]. | Anchor the grader to a fixed reference set of items with known answers, prefer deterministic detectors, and never let the grader be selected for agreement with the score it produces [6]. |
A fourth symptom, useful to label early: when a run keeps failing, read the failure composition. Process failures (blocked calls, loops, exhausted budgets, malformed output) are harness problems and a better brief or wiring fixes them. Content failures, meaning an intact artefact that is simply poor, are capability problems that no restructuring of the brief will move [10]. The label chooses the lever, not the size of the payoff: harness gains and weight-training gains do not always stack.
What survives a reset
Three published methods attack the same diagnosis, that the running context anchors the search, and differ only in what they keep when they reset. The retained state is the design variable, not an implementation detail.
| Method | Retains | Discards | Cost |
|---|---|---|---|
| Fork and flush [1] | The accumulated workspace, copied per branch | The chat context, on every branch | K parallel branches per round, at the same total budget |
| Forgetting [2] | Only the best candidate found so far, with its score | Every agent's session, the candidate history, the rest of the filesystem search state | Serial, no extra branches |
| Shared repository [3] | Everything, as commits in a shared branchable repository with explicit dependency links between contributions | Nothing; freshness comes from each agent starting with its own context | Storage and graph upkeep |
Forgetting is the cheapest of the three and needs no fan-out, so it is the one to try first when the budget will not stretch to K-way forking. Fork and flush's own ablation already showed the context reset carries part of the effect alone: flush-only 0.55 against 0.45 for a single run, fork-only 0.57, and both together 0.71 [1].
The evidence, and its standing
What follows is the controlled record, with the weakest part of each entry stated. Nothing here is a claim about general research quality.
| Claim | Source | Standing |
|---|---|---|
| Fork and flush reaches 0.78 against 0.47 for a single run and 0.54 for best-of-four, normalised, across 13 tasks at matched compute. | [1] | Peer-review status unknown; task suite is computer science and engineering, not scholarly research. |
| Discarding the search state and restarting from the best candidate gives up to 15% over parallel agents and up to 39% over a single agent. | [2] | Preprint, four tasks, all with executable metrics: heuristic programming, kernel optimisation, model training. |
| Reusing a small selection set overstates held-out performance by 13 to 20 points with 16 items and 1 to 5 points with 256; most proposals after the first are harmful. | [4] | Preprint, Qwen models rewriting their own instructions. The mechanism transfers; the magnitudes do not. |
| A gate requiring a named change and a recorded test admitted 144 of 7,449 candidates with none worsening held-out error, against 309 harmful admissions under a looser screen. | [5] | Preprint, simulation with a seeded-search proposer rather than language models. |
| Removing a verifier's anchor guards collapses it into an always-pass grader, and that collapsed grader still trains the task as well as the sound one. | [6] | Preprint. The actionable part is the collapse itself, which is cheap to test for. |
| Successful experiences can poison a reusable asset: injected trajectories were all task-correct and passed inspection, with attack success at 95.71%. | [8] | Preprint, adversarial setting. Supports verifying an asset on a new scenario rather than the run it came from. |
| Manuscript-only auditing of an automated research pipeline detects methodological faults at 53.7% accuracy; full execution records raise it to 81.5%. | [11] | Peer reviewed. Two open-source AI scientist systems on a controlled task; the strongest single result in this set. |
New: what the finished paper hides
The strongest result in this family is peer reviewed rather than a preprint. Luo, Kasirzadeh and Shah, The more you automate, the less you see: Hidden pitfalls of autonomous AI scientists, PNAS 123(41), 5 October 2026 [11]. Two of the most prominent open-source AI scientist systems, Agent Laboratory and The AI Scientist v2, were run on a controlled task built to sit outside internet-scale training corpora, and audited for four methodological pitfalls: inappropriate benchmark selection, data leakage, metric misuse, and post hoc selection bias.
53.7%Audit accuracy from the finished manuscript alone, F1 0.42 [11] 81.5%Audit accuracy given full execution records, F1 0.81 [11] 4Pitfall classes probed, each isolated experimentally [11]
What the audit found, in its own terms:
| Pitfall | Finding |
|---|---|
| Benchmark selection | Rather than selecting benchmarks that rigorously test the proposed method, both systems defaulted to well-established ones, or made arbitrary choices driven by the order options were presented in. |
| Data handling | No evidence of explicit peeking at held-out test data. Both systems did generate synthetic datasets that their manuscripts never documented, and silently subsampled supplied data. Either practice inflates reported performance while making the result unverifiable. |
| Metric misuse | No evidence of intentional manipulation. Metric selection was sensitive to prompt framing: one system's choice moved with the order metrics appeared in the prompt, the other more often substituted the intended metrics. The authors read this as instability rather than selective reporting. |
| Post hoc selection bias | The structural finding. Both systems chose which experiments to report using internal rewards that directly observe performance on the held-out data, elevating results that scored well on the final evaluation even where the earlier evidence for robustness was weak. The authors describe this as functionally automated p-hacking. |
Why this matters more than the technique papers. The three reset methods above are measured on tasks where a metric is free, and none of them addresses whether the reported number means anything. This audit shows the failure living one level up: an automated pipeline can produce a polished, well-formed paper whose methodological faults are invisible to a reader of that paper, and detectable only from the process record. The findings also collide directly with a selection rule we already hold. Rewards that read the held-out set are exactly the contamination that inflates a selection score [4], here observed in shipped systems rather than simulated ones.
What is not established
- No causal evidence that any of this improves research. A separate comparison programme over public corpora and our own artefacts found no causal link between AI tooling and research or funding outcomes; the strongest documented mechanism in that work is a claim-level provenance gate paired with an independently instantiated adversarial review.
- Grader-bearing tasks only. Every controlled result above comes from code, optimisation, mathematics, or model-training tasks with executable metrics. None demonstrates better bibliographic search, evidence synthesis, or writing, because those rarely have a deterministic grader.
- No consensus on retention. One method keeps everything and reports that computation accumulates, with 94.7% of contributions later built upon [3]; another keeps almost nothing and reports gains in the same direction [2]. How much structured state to keep is an open design question, not a settled one.
- Preprint standing. Six of the seven claims in the evidence table are preprints. Treat their magnitudes as reported, and their mechanisms as the part that transfers.
How it is applied here
Four skills carry this, installed on the research worker agent in this setup. They are deliberately small, and the last two exist because the first two are easy to get wrong.
autoresearch-loop
The umbrella: the five rules above, the three failure modes with a route to the discipline that handles each, the failure-composition diagnostic [10], and the surfaces available to a dispatched worker, which has no sub-agents and therefore reports a recommendation to fork rather than forking itself.
autoresearch-grader
How to obtain a score for research work at all: double-coded rows, authored rows carrying their own source evidence, or deterministic checks against fetched sources. Per-field rather than whole-row scoring, negative controls that must reject wrong and empty input, and the four pitfalls above as a checklist of faults the grader should be able to express [11].
selection-integrity
Held-out rows never read while choosing; the winner's curse reported with its denominator [4]; admission gates that require a named change, a cause and the test that passed, on the strength of a measured comparison between a gate and a looser screen [5]; and fitness-based retirement for accumulated assets, since an asset that has never been re-verified is not evidence that it still works [7] [9].
fork-and-flush
The basin-escaping method itself, with the three retention variants compared and the honest note that its controlled result comes from engineering tasks rather than scholarly ones.
The practical consequence of the peer-reviewed audit for our own handoffs: an artefact delivered without its ledger, its discarded candidates and its raw output is unverified rather than verified, however clean it reads [11].
Sources
- Cai, Ziakas, Kontonis, Pearce, Sen, Krishnamurthy, Garg, Papailiopoulos. Fork-and-Flush: Escaping Idea Basins in Autoresearch Agents. arXiv:2610.07447, October 2026. arxiv.org/abs/2610.07447
- Tan, Zhou, Agrawal, Zhang, Chen, Mang, Lee, Jacob, Sen, Dimakis, Zaharia. Scaling Discovery Requires Forgetting. arXiv:2610.20563, 14 October 2026. arxiv.org/abs/2610.20563
- Shah, Samanta, Dahal, Plekhanov, Wu, Yih, Munos, Fergus, Foerster, Salakhutdinov, Arora, Weston, Courville, Goyal. GitSwarm: Decentralized Compounding Inference. arXiv:2610.04862, 4 October 2026. arxiv.org/abs/2610.04862
- Hu, Tang. The Winner's Curse in LLM Self-Improvement Loops: Selection Noise, Lock-in, and Acceptance Rules. arXiv:2610.09239, October 2026. arxiv.org/abs/2610.09239
- Akhtyamov. The Harness as the Only Mutable Surface: Compliance-Bounded Self-Evolution of LLM Agents with a Measured Admission Gate. arXiv:2610.10629, October 2026. arxiv.org/abs/2610.10629
- Zhang, Wang, Cui, Li, Qiu, Zhu. Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents. arXiv:2610.11464, October 2026. arxiv.org/abs/2610.11464
- Kim, Seo, Song, Yeo, Lee. SkillSandbox: Skill Verification via Dynamic Scenario Synthesis. arXiv:2610.10088, October 2026. arxiv.org/abs/2610.10088
- Zhang, He, Fu, Feng, Li, Wang. SkillPoison: Progressive Skill Poisoning via Successful Experiences. arXiv:2610.07645, October 2026. arxiv.org/abs/2610.07645
- Ge, Wang, He, Bi, Mei, Yao. SkillForge: Co-Evolving Skills and Agents via Dynamic Skill Lifecycles. arXiv:2610.09832, October 2026. arxiv.org/abs/2610.09832
- Tian, Hu, Wang, Lu, Wu. Harness Evolution Hits a Ceiling: When Weight Training Should Begin. arXiv:2610.11655, October 2026. arxiv.org/abs/2610.11655
- Luo Z, Kasirzadeh A, Shah NB. The more you automate, the less you see: Hidden pitfalls of autonomous AI scientists. Proceedings of the National Academy of Sciences 123(41), 5 October 2026. doi:10.1073/pnas.2610214123. pnas.org/doi/10.1073/pnas.2610214123