What an ant colony taught us about teams of AI models
We put several different language models into one simulated colony to see if the mix would beat its best member. The answer pointed somewhere more useful.
· 2 min read · research · antelligence
In Antelligence, Kashyap and I build swarms of simple agents that coordinate the way ant colonies do. This fall we asked a question that a lot of people in AI assume the answer to: if you put several different language models into the same colony, does the mix do better than its best single member?
How we set it up
We wrote the hypothesis and the success criteria down before running anything. That is called preregistering, and it keeps you honest: you can’t quietly move the goalposts once you’ve seen the results.
We also gave every setup the same number of model calls. Without that rule, a bigger team can look smarter just because it got to think more. Many “teams beat individuals” results are really “more compute beats less compute”.
What we found
The mixed colony matched its best member. It did not pull ahead of it.
That sent us back to an earlier benchmark, where the colony had shown a clear gain on planning tasks. When we took that result apart, the gain came mostly from a simple merge step in our engine, the part that combines the colony’s plans without using any model at all. Teaming up small models didn’t add much on question answering either.
In other words, the colony’s strength wasn’t coming from mixing models. It was coming from how the colony combines its work. That is a much more useful thing to know, because it tells us exactly where to build next.
Why this is the result I trust most
- It was decided in advance. The plan said what would count, so there was no room to talk ourselves into a story.
- The comparison was fair. Matching compute removed the easiest way to fool ourselves.
- It explained something. It showed us which part of the system actually does the work.
My honesty-steering research, Honest to a Fault, taught me a similar lesson. Finding a signal inside a model is one thing. Being able to use it as a reliable lever is another, and only careful tests tell the two apart.
What we’re carrying forward
- Credit the simplest component first. If a system improves, check whether the gain survives with the clever part removed.
- Match compute in every comparison.
- Share what you find, whichever way it points. It saves the next person from repeating the experiment.
The colony stays, and the merge step just became one of the most interesting parts of it. That’s where Antelligence goes next.