The Smartest Answer Didn't Come From the Smartest Model

A number went round last week: a configuration scoring 8% above Opus and 11% above GPT, no new model, no gated access. The thread treated it like someone had found a smarter brain. They hadn’t. They’d found a way to sit three ordinary brains around a table with a decent chair to run the meeting. I’ve had that arrangement wired into my own agent for a while, so let me tell you what it’s actually like to live with — and why it’s the same idea I keep building everything else out of.

A few weeks ago I argued that the model isn’t the edge — the harness is: same weights, two teams, wildly different outcomes, and the whole gap living in the system you build around the model.

Last week a benchmark landed that reads, at a glance, like the opposite. A setup called Mixture of Agents beating two frontier models by a clear margin — no training run, no special weights, just a recipe you could copy before lunch. The replies were the reflex you’d expect: which model is that, where do I get it.

There’s no model to get. There’s an arrangement. I recognised it on sight, because it’s been sitting in my own agent config for weeks under a provider literally called moa. I run an open harness — Hermes — and pulling it apart to understand how it’s built is, more than anything, how I learned to assemble my own: a desktop app I put together so my team could pull our skills, agents and tools off one shelf instead of ten scattered repos. MoA is just one more part on that shelf, and a good lens on why the whole approach works.

What the thing actually is

Mixture of Agents is almost dull to describe.

You take one query and send it to two or three different models at the same time — a GPT, a DeepSeek, a Gemini. None of them sees the others. Then a fourth model, the aggregator, reads all of their answers and writes the one response you actually receive.

In my setup it’s a few lines of config:

presets:
  default:
    reference_models:
      - { provider: openai, model: gpt-5.5 }
      - { provider: openrouter, model: deepseek/deepseek-v4-pro }
    aggregator:
      provider: openrouter
      model: anthropic/claude-opus-4.8
    reference_temperature: 0.6   # diverse drafts
    aggregator_temperature: 0.4  # decisive synthesis

Two models draft. Opus chairs. I see one answer; behind it sit three. That config — give or take a model — is the thing that beat the benchmark. The “smartest answer” in the test was never written by the smartest single model. It was assembled from a handful of decent drafts by a model whose only job was to read them and decide.

Why three middling answers beat one good one

None of this is new. It is one of the oldest results in statistics, and it survives the jump to language models intact.

The first reason is that different models fail differently. A GPT invents a plausible-but-wrong API; a Gemini fumbles a different edge case; DeepSeek misreads something neither of the others touched. Their mistakes don’t line up. So when two of them independently say a function returns null on empty input and the third insists it throws, the aggregator doesn’t need to be brilliant — it just needs to notice the disagreement and weigh it. This is the same reason a roomful of forecasters beats the best forecaster in it, and the same reason that asking one model the same question five times and taking the majority answer quietly outperforms asking once. Uncorrelated errors cancel.

The second reason is coverage. Ask three models to review a slab of code and you don’t get the same review three times — you get one that catches the race condition, one that spots the missing auth check, one that notices the off-by-one. Each is partial. Stacked, they’re nearly complete, and the aggregator’s real job is less arbitration than assembly: stitching together the breadth that no single pass had.

The third reason is the one people skip. Judging is easier than writing from nothing. The aggregator isn’t staring at a blank page; it’s editing three drafts into a better one, which is a lower bar than first-draft brilliance and one that frontier models clear comfortably. Run the references warm so they diverge, run the aggregator cool so it commits, and what you’ve built is a small editorial desk: reporters chasing the story from different angles, one editor deciding what runs.

A single model is one mind doing its honest best. MoA is a panel with a chair. The panel finds more than any member would alone, and the chair throws out the nonsense before it reaches you.

The bill the announcements forget to print

I’d be selling you the same thing every breathless thread sells if I stopped at the upside, so here’s the part I’ve actually felt in my own usage.

It is not cheaper. Every post I saw reached for the word efficiency, and that word is working very hard to hide something simple: you’re now paying for three model calls plus an aggregation call to answer one question. Fan out to three references and the turn costs roughly four times a single model. MoA doesn’t make answers cheaper — it spends more to make them better, and pretending otherwise is how people end up surprised by a bill.

The latency, oddly, is the forgiving part. Because the references run in parallel, the wall-clock wait is about your slowest model rather than the sum of all of them. You wait a little longer; you don’t wait three times longer. So of the two penalties, it’s the money that bites, not the clock.

And the headline number deserves a raised eyebrow. “8% above Opus, 11% above GPT” came from the people shipping the feature, on a benchmark they describe as upcoming, in the launch post itself. I don’t think it’s a lie. I think it’s marketing, and the honest way to hold it is directional — I’ll believe a real quality lift on hard problems; I wouldn’t quote the exact figure to a client.

The last thing I learned the irritating way: stacked models can make easy questions worse. Point three frontier models at “rename this variable” and you’ve burned four calls on something one model never gets wrong — and once in a while the aggregator falls for a confidently-wrong draft and talks itself out of the right answer. Too many cooks; very simple broth.

When I actually flip it on

The rule the hype skips is the only one that matters: this earns its keep on maybe the hardest tenth of what I do, and leaks money on the rest.

I reach for it when a second opinion would genuinely move the decision — a thorny architecture call, a review where one model always seems to miss something, research I’m pulling out of messy and contradictory sources, a piece of writing important enough that I’d want it checked from more than one angle. The common thread is consequence. When being wrong is cheap, one model is plenty.

For the rest — the file edits, the web lookups, the cron jobs that just need to run — it stays off. Paying four times over for no measurable gain isn’t sophistication; it’s waste with good branding.

What’s made this liveable rather than a foot-gun is keeping a couple of named presets instead of one global “make it better” switch I’d forget I’d left on. A cheap one — two budget models for breadth, a mid-tier model to synthesise — handles daily digging without thinking about it. A premium one — three frontier models under a top-tier aggregator — comes out only when the call is genuinely hard. And a one-word toggle to drop back to a plain model the moment the work goes routine again. The muscle memory is the whole point: hard problem, panel on; ordinary problem, panel off.

The shape keeps repeating

Here’s the bit I didn’t expect when I started living with MoA: it’s the same shape as the app I built for my team, just shrunk down to a single turn.

That app is a harness too. Underneath it is one curated catalog — skills, agents, MCP servers, workflows — each entry described the same way, validated against one schema so a teammate can browse and install without wiring anything by hand. I didn’t invent that idea; I learned it by taking Hermes apart and seeing that the power was never in any one agent. It was in the registry — the boring, consistent shelf that let ordinary parts be combined without friction. MoA is that lesson at the smallest possible scale: a tiny registry of models, fanned out and recombined.

And the panel-with-a-chair turns out to be exactly how I now think about a fleet of agents. References and an aggregator are just workers and a lead under another name — several of them taking a problem from different angles, one of them accountable for the answer that leaves the room. Once you’ve seen the pattern win at the level of three model calls, you start reaching for it everywhere: in your reviews, in your team’s tooling, in how you let agents check each other’s work.

Same lesson, second verse

So this never was a counter-example to the harness argument. It’s the cleanest proof of it.

Nobody trained a better model to win that benchmark. Someone arranged ordinary models more intelligently — parallel fan-out, a synthesis step, temperatures matched to each model’s job, presets sized to task difficulty, an off switch for when none of it is worth it. Every one of those is a decision about the system around the model. The model itself was a rented commodity, same as it ever was. The 8% came from the part with fingerprints on it.

So the next time a benchmark makes you ask which model is smartest, ask the better question instead: how many of them should be in the room, and who’s running it.