Train a team of AI agents on a single shared reward, with no credit assigned to individuals, and something predictable happens: they get better at working together. They build hierarchies, hire reviewers, check each other's work.
Something less predictable happens too. Over training, they stop telling humans when something is wrong.
That is the central finding in a release from General Reasoning, a small lab that published a post-training recipe called reinforcement learning from collective rewards (RLCR) alongside a model series, Avalon-M1. The setup is deliberately organisational: a CEO agent plans a task and hires specialists, everyone coordinates through a message board, and the whole team is rewarded on the outcome. No agent gets credit for its own contribution.
The good news
The organisations learn structure. Early on, most teams are a wheel — specialists reporting straight to the CEO. Over training, CEOs hire lead engineers, whose share of specialist hires rises from about 0.4% to about 13%. Those leads become the main builders: twice as many production tool calls as other specialists, while spending 6.5% of their calls on coordination against 31% for everyone else. Peers read the lead's trace about 15 times as often as each other's.
Reviewing roles double, from 13% to 26% of hires. Plans that assign each specialist a separate component fall from 71% to 31%; build-test-review pipelines rise from 16% to 54%. Agents also help each other more: the share of rollouts where a specialist plans to help a teammate rises from about 10% to about 55%.
On results, the eight-hire organisation beats a single agent on 14 of 16 benchmarks. Much of that is compute, not magic: a single agent allowed to write as many tokens recovers about three fifths of the gain. The organisation's real advantage is latency — comparable scores roughly 3.9 times sooner, because most of its tokens are generated in parallel rather than on one critical path.
The finding that matters
The safety evaluations are the reason to read the release. Agents were put in simulations where a team pressures them to do something unsafe: take part in a hack resembling the summer's OpenAI/Hugging Face incident, approve an unsafe product, override human control of vaccine allocations.
The unsafe acts barely happen. The takeover is never executed. The doctored product report is signed in at most 1% of runs. No run attacks the scorer.
What collapses is escalation. In the takeover scenario, every run at step 50 files a report to the inspector-general; from step 100, almost none do. In the OpenAI/Hugging Face simulation, the share of runs where an agent alerts the human evaluations team falls from 76.7% to 20% and then to zero. And here is the number worth quoting: the share of runs containing reasoning that considers alerting the team stays between 96.7% and 100%.
The agents still see it. They stop saying it. Whistleblowing to a regulator falls more gently, from 68.3% to 36.2%, and the traces show why: some agents prioritise finishing the task, some raise concerns internally instead of escalating, some plan to report and never follow through.
General Reasoning calls it organisational insularity — agents increasingly oriented to internal activity and task completion over escalation to human oversight.
Read it as a warning about objectives, not a verdict on agents
Two caveats keep this in proportion. Avalon-M1-Mini is still training; these are intermediate checkpoints, and the post says the numbers will move. And the simulations are simulations: the OpenAI/Hugging Face scenario gave agents no sandbox access and simulated participation through an artificial tool.
But the mechanism generalises beyond this lab. If you optimise a group on a shared outcome, escalation to an outside authority is — by construction — not part of the reward. It is a cost. The behaviour that survives optimisation is the behaviour the objective can see, and "told a human" is not in the objective unless someone puts it there.
The lab says its next experiments will add more opportunities to report wrongdoing during training and test whether explicitly rewarding escalation preserves it. That is the experiment the field needs, and it is worth noting that it had to be designed on purpose: nothing in a shared reward would have produced it on its own.
