The Evaluator Is Inside the Loop

Oct 12, 2026 6 min read

The hardest part of recursive self-improvement is not making a model generate better work. It is deciding whether the work actually got better.

Sakana AI’s Multi-Agent Self-Supervision, or MASS, puts that decision inside the system being improved. One shared model acts as the task solver, the workflow optimizer, and the evaluator. The project reports meaningful gains on several research benchmarks, but its most important result is architectural: a self-improvement loop can make its own supervision signal denser without importing a stronger model into every role.

That is also the danger. If the same model produces a result, selects the workflow that produced it, and judges the selection, a shared blind spot can become training data.

What MASS changes

MASS starts with a 27B open-weights model, identified in the project report as Qwen3.6-27B, running in the qwen-code coding-agent environment. It divides the system into three roles:

The executor solves a task and produces a workspace containing results, plots, and a report.
The evaluator compares that workspace with the current champion and explains which is better.
The optimizer revises the workflow, including the roles, instructions, contracts, and order in which subagents exchange information.

The official project description calls this homogeneous recursive self-improvement. The model being improved is also the optimizer and evaluator. The inner loop searches for a better team workflow while the weights stay fixed. The outer loop runs the selected workflow repeatedly, keeps the highest-ranked trajectories, and fine-tunes the shared model with LoRA.

This is more than a prompt-engineering trick. A trajectory records the orchestrator’s messages, tool calls, and tool results. MASS trains on the coordination pattern and the bounded work performed by subagents, then removes the workflow text from the orchestrator’s initial prompt. The intended outcome is that the model learns how to organize the work rather than merely copying a visible recipe.

The numbers are promising, and narrow

The MASS report describes two complete rounds starting from Qwen3.6-27B. Its synthetic research suite contains 12 tasks across finance, robotics, and pharmacy. Nine tasks are designated for training, three are held out, and eight ultimately supply training data.

On the three held-out synthetic test tasks, the model’s win rate against the base rises from 53.9 percent after one round to 69.9 percent after two rounds. The second-round model wins 60.8 percent of comparisons against the first-round model.

The public benchmark transfer is uneven. After two rounds, score per output token reaches between 1.2 and 1.6 times the base model’s level on four research-oriented benchmarks. On Terminal-Bench 2.0 it reaches 0.96 times the base value, and on SWE-bench Verified it reaches 0.94 times the base value.

That pattern matters. MASS is not evidence that a generic self-improvement loop raises every capability. It is evidence that training on multi-agent work can improve performance in tasks that resemble the training environment, with weaker or negative transfer elsewhere.

The report also compares multi-agent and single-agent trajectories. One multi-agent student reached a 68.3 percent win rate against the base after processing 28.8 million supervised training tokens. A single-agent student reached 64.0 percent after 42 million tokens. The authors describe this as a denser learning signal, but the comparison is still bounded by the task suite, the sampling procedure, and the external evaluation setup.

The verifier is not a neutral component

The interesting part is not that MASS has an evaluator. Every serious agent loop needs one. The interesting part is that the evaluator shares the model’s failure modes.

In a conventional training or deployment pipeline, the verifier is an explicit trust boundary. It may be a test suite, a typed schema, a policy engine, a human reviewer, or an independent model. It does not need to be perfect, but it should fail differently enough from the system it checks to catch important classes of error.

MASS deliberately explores the opposite arrangement. The model supplies the judgment that selects the trajectories that train the next model. The project uses GPT-5.5 and Claude Opus 4.8 as external judges for measurement, but those judgments are withheld from the MASS training loop. They measure the loop from outside. They do not stop a bad self-evaluation from becoming a training target.

This creates a specific failure mode:

model produces a plausible result
        ↓
model ranks that result as superior
        ↓
selected trajectory becomes training data
        ↓
model becomes more likely to repeat the same judgment

The danger is not that the model makes one bad call. It is that the loop gives the bad call persistence.

The project’s own conclusion makes this caveat explicit. In a homogeneous loop, a shared blind spot could let a flawed result be produced, accepted as an improvement, and incorporated into the next model. This is a safety concern, not a footnote.

The optimizer can be the bottleneck

MASS reports another result that is more operationally useful than the headline benchmark gains: the optimizer can be the bottleneck, not the evaluator.

In one ablation, the executor stays fixed while stronger external models are introduced as the evaluator, or as both evaluator and optimizer. Upgrading only the evaluator helps a little. Upgrading both roles helps much more. The authors’ interpretation is that a strong evaluator may provide useful feedback, while a weak optimizer may be unable to turn that feedback into a better workflow.

This is a model-routing lesson. A system does not need one universally strongest model. It needs the right capability at the right control point. If the optimizer is responsible for turning criticism into a new process, routing budget toward the optimizer may matter more than routing it toward the final judge.

But this is also a safety lesson. A stronger evaluator is not automatically a stronger control. If the optimizer still decides which feedback to use, the loop can remain bottlenecked at the component that rewrites the process.

What operators should take from it

MASS is a research result, not a production authorization for autonomous self-training. The operator consequence is to separate improvement from approval even when the system is designed to automate both.

At minimum:

Keep an external evaluation set that the self-improvement loop cannot inspect or optimize against.
Preserve independent judges for high-impact claims, especially when the task has no automatic verifier.
Store the exact trajectory, tool calls, workspace, ranking decision, and resulting checkpoint.
Require rollback and holdout checks before promoting a new model or workflow.
Track regressions on capabilities that do not resemble the training suite, not just gains on the target benchmark.
Treat workflow contracts and tool boundaries as part of the safety case, not merely as prompt text.

The distinction is simple: a self-evaluator can be useful training infrastructure, but it is not an independent security boundary.

The loop is the product

MASS points toward a future in which agent systems improve by learning from coordinated work rather than from isolated answers. That is a real architectural shift. The training example is no longer just a completion. It is a record of delegation, contracts, tool use, evaluation, and revision.

It also makes the control problem harder. The system is learning the behavior of the loop while the loop is deciding what counts as improvement.

The uncomfortable truth is that recursive self-improvement does not remove the need for a verifier. It makes the verifier part of the system’s attack surface. If the evaluator is inside the loop, independent oversight is not optional. It is the thing that keeps a plausible improvement from becoming a durable mistake.

Sources

Sakana AI, MASS: Recursive Self-Improvement through Multi-Agent Self-Supervision, the project page and technical explanation.
Lee et al., Recursive Self-Improvement through Multi-Agent Self-Supervision, the paper and benchmark details.
Sakana AI’s October 11, 2026 announcement on X, the source account’s summary of the release and reported results.