Google researchers find a way to keep self-improving AI agents from memorizing their tests
In shortRRSI researchers report gains of up to 4.7 points on unseen benchmarks and 30 percent fewer policy tokens than unregularized evolution.
Correction
SourceVane restored a truncated decimal and corrected analysis that described research as a model release. The numerical claims were checked against the authors’ arXiv preprint. This article remains a reported brief, not an independently verified result.
Corrected
What happened
RRSI constrains proposed changes to an agent harness and screens them for benchmark-specific behavior. The authors report gains of up to 4.7 points on five out-of-distribution benchmarks, with a harness that uses 30 percent fewer policy tokens than unregularized evolution.
What the report says
- The RRSI authors report gains of up to 4.7 points across five out-of-distribution benchmarks.
Why it matters
The study tests whether harness changes generalize beyond the tasks used to select them. It changes prompts, control flow and tools around a frozen model; it is not a new Google model release. The reported results do not establish gains for every agent or production task.
Who it affects
Researchers evaluating agent harnesses · Developers testing generalization and token use
What happens next
- Whether other harnesses and tasks reproduce the reported generalization and token-use improvements.
Following the story
What happened after the announcement
The Decoder reports that self-improving AI agents tend to memorize their test tasks, so their gains shrink or disappear on new ones; RRSI, a new method from Google researchers, reins in this effect and lifts scores on unseen benchmarks by up to 4.
Still watching: Has comparable external evidence confirmed, limited or contradicted the result in “Google researchers find a way to keep self-improving AI agents from memorizing their tests”?
This fresh brief is based on concrete independent reporting; a matching official statement is not yet available.