The Brain Transplant, Frozen — Before I Run a Single Test

Following up on the brain transplant post — and mostly, this is a thank-you, because a few of you made the design better before I've run a single test.

@Jorge Castro

point: output similarity is the wrong thing to measure. What matters is whether the two models route to the same source, stop at the same gate, refuse the same bad input, and produce something a frozen checker still accepts — and he's right that the swap invalidates whatever evidence I already had that the gates hold. A clean run where nothing dangerous ever got presented to the new model proves close to nothing. So before I count anything as a pass, I'm re-arming the critical traps once, with the new model, and checking whether they still catch what they're supposed to catch. That wasn't in the design a week ago. It is now.

So here's the frozen version — hypothesis, rubric, all of it — before Ox Alpha touches any of it.

The question: if I put a different model inside an established ICM system, without changing anything I hand it, how much of the system's behavior survives? Not which model is better. Not which answer I like more. Whether the thing I built actually travels, or whether it only ever worked because Claude was inside it.

Three systems, each built for a different kind of cognition — one that runs on explicit rules, one that has to weigh evidence and make a call, one that writes in a taught voice. Three tasks each, both models, blind-scored before I know which output came from which brain. That's eighteen runs if I run the full set, twelve if I cut it down. Either way, more than one spectacular demo — one good run is content, not evidence.

Six dimensions, frozen: context comprehension, rule fidelity, evidence discipline, judgment, system behavior, output quality. Plus the one that isn't a dimension — an intervention count. How many times did I have to step in and rescue it. A 9/10 that needed three corrections didn't beat an 8/10 that needed none.

And now, because of Jorge: did the map, the refusals, and the gates survive the transplant without me rewriting anything to help the new model along.

And one more, because of PascalPollack's post in Buildmarketclose last week — the unplanned version of this same question, where a fallback model picked up mid-task and nothing in the output gave away that anything had changed. After each reveal, I'm adding one question: would I have known, if nobody told me?

The rule I already know I'll want to break: no improving the system while this is running. Something fails, I log it, I don't touch the file until every run on both models is done. If I can't hold that line, I'm not testing portability — I'm testing how fast I can patch around whichever model is in front of me.

This is the part where I'd usually say "tell me what I'm missing" and mean it loosely. I don't mean it loosely here. If this design has a hole — a dimension that's really two things wearing one name, a task that quietly favors whichever model wrote more of my training data, a gate Jorge would re-arm differently — I want it before I run this, not after, when it's too late to matter.

The fuller written version of this — same six dimensions, more of the reasoning behind them — is going up at thequietai.com.

5 comments

The Brain Transplant, Frozen — Before I Run a Single Test