I Blogged That the Chair Was a Genius. Then I Built It as a Bouncer.

I built the AI judge I’d just spent a blog post praising. The first thing I deleted was the judge. In its place went a deterministic gate: fail your build, your tests, your lint, or a policy rule, and no model gets paid to read your draft. Two months ago I called the chair the smartest model in the room. Building it taught me the chair earns its keep by being cheap and objective first — and brilliant only when it has to be. This post corrects the last one.

In June I published a piece on Mixture of Agents: fan a query out to a few models, let a strong aggregator read every draft and synthesise the answer. A panel with a chair. I called the chair the room’s best mind and left it there.

Then I built the thing. Not a prompt — an actual contest coordinator wired into the fleet that runs my day-to-day: coding agents, reviewers, research agents, each backed by real CI. One task fans out to N makers, and a judge decides what survives. And building it did what building always does to a blog post written before the code. It found the two places I’d waved my hands.

Here’s what I got wrong, what the code taught me, and — because I’d rather you learn it from me than from a bill — the brutal review of my own earlier post at the end.

The shape, end to end, is five steps, and only the last two can involve another model:

makers fan out  →  deterministic gate (fail-closed)  →  rank survivors on a held-out rubric
                →  select one OR assemble all  →  escalate ties to a model or a human

By design, the goal is for many contests to avoid a second model entirely — the gate is meant to settle whatever it can. Here’s why that matters.

Mistake one: I made the chair read everything

In the MoA post, the chair is an LLM. Every answer you receive has been paid for four times: three drafts, plus a frontier model reading all three to synthesise. I even wrote the reassuring line — “judging is easier than writing” — as if that settled the cost question.

It doesn’t. “Easier than writing” is still “pay a frontier model to read N drafts, every single turn.” And here’s the categorical mistake underneath it: a losing draft often isn’t worse than the winner — it’s invalid. It doesn’t compile. Its tests fail. It trips a policy rule. Invalidity isn’t a judgment call the way “which of these two working solutions is better” is; it’s a fact you can check. Yet in the blog’s design, that broken draft still gets read by the expensive chair before anyone notices it never had a chance. Validity and quality are different contracts, and I’d handed both to the same overqualified model.

The fix, once you see it, is almost embarrassing. Often, “which draft is best?” is secretly “which drafts are even valid?” — and validity is something you can check cheaply, deterministically, before any model is paid to have an opinion.

So the judge I built leads with a gate, not a mind:

# Illustrative — the gate is a pure function, no model calls.
GATE = ("build_passed", "tests_passed", "lint_passed", "policy_passed")

def passes_gate(candidate) -> bool:
    # fail-closed: a MISSING signal counts as a failure, never a pass
    return all(candidate.gate.get(check) is True for check in GATE)

The gate runs first on the CI you were already paying for. It isn’t free — CI has latency, flake, and blind spots — but against a frontier-model read it’s cheaper, more reproducible, and more objective, which is the only bar it has to clear. Drafts that fail it are out before the chair convenes. In a contest where, say, three of five drafts don’t compile, the expensive model reads two, not five. The chair stopped being the room’s smartest mind and became its bouncer — and the bouncer barely costs a thing.

The is True is doing real work there. Fail-closed means a missing signal is a failure, not a pass — a maker can’t sneak through by simply not reporting a check. And you need a third state beyond pass/fail: inconclusive. A test that times out, a flaky suite, a policy check that couldn’t run — those aren’t “invalid,” they’re “unknown,” and treating unknown as pass is how a broken draft reaches the chair anyway. My gate routes inconclusive candidates to a re-run or an escalation, never to the winners’ pile. The gate is only as honest as its willingness to say “I couldn’t tell.”

Why this generalises: the naive LLM-as-judge and Best-of-N setups pay a model to read every candidate. If any part of “good” is objectively testable — it runs, it passes, it doesn’t violate a rule — that test belongs before the judge, not inside it. Verification is cheap; judgment is expensive; put the cheap thing first. This is the deeper rule the blog missed: deterministic systems decide admissibility; a model only arbitrates among the admissible. Validity is an admission ticket, not a ranking criterion — the moment you stop treating it as one, the architecture changes. That reorder changed when I use a panel at all. Before, I reserved them for high-stakes calls. Once the gate started ejecting invalid drafts before any model read them, I could justify a panel on ordinary coding and review work too. The first time the gate killed a broken diff before any model saw it, I understood the judge’s main job wasn’t intelligence — it was refusal.

Mistake two: I blurred two different tools into one word

The MoA post has one verb for what the chair does: synthesise. Read the three drafts, write the better one. Assembly.

But when I sat down to implement it, the tasks refused to agree on what “the answer” even was.

Some tasks have one artifact that must land — a code change, a single decision. You don’t want a Frankenstein diff stitched from three makers; you want the best one, whole and coherent. That’s Best-of-N: rank the survivors, pick one, discard the rest.

Other tasks are about coverage — a code review, a research pass. Here, throwing away the losers is the bug. One reviewer caught the race condition, another the missing auth check, a third the off-by-one. The value is the union. That’s Mixture of Agents: keep every catch, synthesise them into one.

My blog had quietly assumed every task was the second kind. The build forced the split into the open, as an explicit mode on the contest:

if mode == "assemble":            # MoA — coverage: keep every catch
    return synthesise(ranked_passers)
else:                             # Best-of-N — one artifact must land
    return select_top(ranked_passers)

Same panel, same gate, same rubric — one flag that decides whether the chair picks or merges. Get it wrong in either direction and you have a real defect: assemble a diff and you ship a monster; select a review and you throw away the auth bug the winner happened to miss.

The rule I use now, decided from the deliverable before a single model is chosen:

• one mergeable artifact, coherence matters → select the best, discard the rest.
• independent findings, recall matters → assemble all, keep every catch.
• mixed task → split it into sub-artifacts, then apply the rule per part.
• unsafe to merge blindly → select or escalate; never auto-assemble.

Why this generalises: “let the models vote” and “let a model synthesise” are marketed as the same idea and they are not. Pick your mode from the shape of the deliverable — one-thing-that-lands vs. everything-must-be-caught — before you pick your models. I’d been calling both “the aggregator” and hoping the distinction didn’t matter. It does: that one flag decides whether you preserve coherence or preserve recall, and choosing wrong silently breaks the task.

Two smaller things the code insisted on

Hold the rubric out from the makers. The chair ranks gate-passers on a weighted score — correctness, tests added, grounding, diff size, risk. The instant a maker can see that rubric, it optimises the rubric instead of the task: pad the tests, trim the diff, game the number. Goodhart’s law with a paycheck. So the scoring rubric is held out from the agents doing the work — they’re judged on a ruler they never get to hold. And the weights only ever break ties among valid candidates — hard disqualifiers live in the gate, not the score — so I log every component of every score, because the day a weight is wrong, that log is the only way to find it.

A tie is not an answer. When the top two survivors land inside a configured tie band — a small score margin I set per contest — the honest output isn’t a coin-flipped winner; it’s “I can’t separate these.” The judge escalates: to a tie-breaking model, or to me. Refusing to manufacture confidence is a feature. A judge that always returns a winner is telling you it’s certain even when the drafts say it shouldn’t be.

Both come from the same instinct: fail closed. A missing gate signal is a failure, not a pass. A tie is a question, not a verdict. When in doubt, deny and surface it — don’t quietly degrade to a confident guess.

The chair’s constitution

Collapse all of it and the judge runs on four rules, cheapest first:

1. Verify before you judge. Anything objectively testable — it builds, it passes, it’s in policy — is settled by a deterministic gate before a model is paid to read a word.
2. Judge before you synthesise. Rank the survivors on a rubric the makers can’t see. No mind touches a draft that didn’t clear the gate.
3. Pick your mode from the deliverable. One artifact that must land → select the best. Coverage that must be complete → assemble them all.
4. Refuse to fake confidence. Ties and empty fields escalate to a human or a tie-breaker. Fail closed, every time.

Notice what the smartest model does in that constitution: by design, almost nothing. It sits on rung three and waits for the case where cheap objective tests genuinely can’t separate two good drafts. That’s the whole inversion. My blog put the frontier model at the centre of the room. The build put it on call.

Brutal review of my own earlier post

I’d be doing the exact thing I criticised — selling the upside and skipping the bill — if I didn’t turn the knife on the original piece. Scored honestly:

What the June post claimed	What building it revealed
The chair is the smartest model, reading every draft.	The chair sits behind a cheap deterministic gate; the smart model is a last resort, not the default reader of every draft.
”Judging is easier than writing” settles the cost.	Judging still pays per draft. Verifying is what’s actually cheap — and it comes first.
The aggregator “synthesises” the answer.	Two different jobs — select one vs assemble all — that I’d merged into one word.
Diversity means diverse models.	Diversity is really across the makers/agents; the models are one knob on it.
Presets sized to task difficulty.	Right — but the bigger lever is the gate, which cuts the cost of any invalid draft regardless of preset.

Where the post holds up: the error-decorrelation argument, the “panel finds more than any member” intuition, the off-switch for routine work, and the core metaphor — a panel with a chair. The bones were right. What I got wrong was assuming the chair earns its keep by being brilliant. It earns its keep by being cheap and objective first, brilliant only when it has to be. You can’t see that from the config file. You see it the first time a broken draft costs you a frontier-model read for nothing.

What this unlocks — and the standing cost

Put the gate first and a panel stops being a thing you ration. Invalid drafts now exit without a model read, so you can afford to convene the room on work you’d never have justified when every turn meant reading three drafts plus an aggregation. The expensive chair only sits when the cheap tests can’t tell two good answers apart, which is exactly when a second opinion was worth buying anyway.

The bill nobody prints on this idea either: your test suite is now management. Your CI, your tests, your policy checks are no longer just guarding main — they’re the first judge in your fleet, the one that runs on every candidate before any model does. Weak tests route bad work downstream; flaky tests erase good work upstream; every blind spot in them is a bad draft your chair now pays to catch. And “valid” and “best” have become two separate contracts, so every task has to declare its answer shape — select or assemble — before the fleet starts. That’s the standing cost, and unlike the benchmark numbers in the last post, it’s one I can actually show you.

So the next time you reach for an AI judge, don’t start by choosing the smartest model. Start by asking what you can prove cheaply and objectively — and only pay for judgment on what’s left. The next judge you build shouldn’t start life as a genius. It should start as a bouncer.

---

This is a correction to The Smartest Answer Didn’t Come From the Smartest Model. I’m leaving that one up, warts and all — the manifesto was right about the shape and wrong about the chair, and the gap between them is the most useful thing in either post.