The Judge and the Oracle: Why a Model Cannot Outgrow What Scores It

Ask a language model what the capital of France is. Then ask it to describe Gödel’s incompleteness theorem.

The first question needs no thought. The answer is memorised, the model has almost no chance of getting it wrong, and no amount of extra computation would improve it. The second is a different object. The model can certainly recite what it has stored on the subject. But given the complexity, it might need a little time to check that each step of what it is saying holds, and that what it delivers is correct.

Here is the pact I suspect most of us would take. We would all rather wait a minute or two for an answer than get one in under a second and have it be wrong. And if we imagine a model helping us with a problem humanity has not solved, which I take to be one of the objectives behind the term general intelligence, then the pact gets larger. We would be willing to say: here is a month of compute to think about this problem and explore what is possible, provided that at the end of it you have solved it.

So the question becomes concrete. How do you let a model think about a problem for longer?

Hallucination as a resource

Whatever the mechanism turns out to be, it needs a process of exploration: generate possibilities, check whether they hold, and see whether they bring you closer to a solution.

Which leads somewhere counterintuitive. What that process needs is hallucination.

Hallucination is the thing everyone works to minimise when putting a model into production, and rightly so. But if you want a model that reasons well, that explores possibilities in order to converge on a solution, then paradoxically you want it to hallucinate. The two regimes are worth separating cleanly, because this is where the idea either lands or does not. There is a phase where you generate, and a phase where you select. In production, a hallucination is an answer, and it is a defect. Inside an exploration, the same hallucination is a hypothesis, and it is the raw material. The word does not name two phenomena. It names one phenomenon at two different points in the pipeline.

This is, after all, a cognitive process we use ourselves. We generate possibilities, most of them bad, then select among them, confront them with the real world, and use what survives as the solution to the problem we were trying to solve.

Think about what this looks like for a coding task. We would not want the model to intuitively spit out a plausible solution. We would want it to say: hold on, let me check the steps I am proposing. I have my own simulation on my side, my own code editor, my own way of compiling, and I can verify, why not with a unit test, that this actually works. And then, at the end, I can give you my reasoning, or just the answer.

Hold on to that example. It is doing more work than it appears to.

Supervising the process, not the output

There is a paper by OpenAI researchers, Let’s Verify Step by Step, that gives the mechanism a shape.

The setup: a first model generates a series of reasoning steps that converge on a final answer, and a reward model examines what was produced in order to judge it. The whole question is what the reward model looks at. It can look only at the output and say whether the final answer is correct: that is an outcome-supervised reward model, an ORM. Or it can have an opinion on each intermediate step, this step of reasoning is sound, this one too, and the answer at the end follows: that is a process-supervised reward model, a PRM.

The paper’s conclusion is that the PRM tends to beat the ORM, precisely because it has learned to supervise the reasoning process rather than to grade a result.

But I think there is a more important conclusion in that paper, and it is more or less hidden. It was not the main point of attention. Once you have a reward model capable of supervising reasoning, the reward model stops being a filter you apply at inference time. It becomes something to train against. The generative model can train itself, with the reward model as its feedback signal, until it satisfies the judge.

That is the jump. Not a selection device: a learning loop. And that jump is the only reason I would call any of this an ingredient for general intelligence rather than a decoding trick.

You can push it further in your head. A generator that produces an enormous number of candidate reasonings, a PRM that prunes them step by step, and only the best surviving solution delivered to the person who asked. I would not be surprised to see some version of this in whatever OpenAI ships next, in a 4.5 or a 5. I have no crystal ball, and that is not the interesting part.

The wall that makes this detour necessary

Why go looking for extra computation at the output, rather than simply training on more data?

I talked about this a month ago, in an episode on the Bitter Lesson, so I will keep it short. The Chinchilla work looks at three quantities together: available compute, number of parameters, and size of the dataset. What it says is that these have to stay in balance. Too little data for the architecture and you underuse your parameters; too much data for the compute and you cannot find the patterns in it. You cannot grow one term without the others.

Now, the Bitter Lesson asks for methods simple enough to keep benefiting from the growth of compute. Fine. But of those three quantities, the one that runs out first is not compute. It is human text.

And here is the observation that convinced me the problem lies elsewhere. Given the size of the databases in use today, we are close to having collected the whole of human text. That corpus should already contain all the information a model needs to understand at least part of how the world works. And yet, when you use these models, you still hit a real ceiling in reasoning. Certain chains of thought simply do not happen, even though they ought to be in the data.

So the problem is not what goes in. It is what happens on the way out. Which is what the previous section was really answering.

One aside, and I hope it stays an aside: as models publish more of their own output onto the internet, that output starts finding its way back into training sets. A model poisoning itself with what it wrote. We will see how much of a problem that turns out to be.

The limit

Now the part I keep coming back to, because I do not think it has a solution yet.

The reward model learns from human annotations. Which means two things, and the second follows from the first. It has no way of progressing on its own. And therefore the generative model that trains against it cannot surpass it either. It can approach it. It can match it. It cannot cross it.

Notice what has actually happened here. The ceiling has not been removed, it has been moved: from the training data to the reasoning annotations. That is a genuine gain, because there is more headroom under the second than under the first. It is not an exit.

The precedent is worth stating plainly, because we have seen this shape before. AlphaGo learned from a database of human games, absorbed the patterns of how a human plays, and reproduced them extremely well. It was only with AlphaGo Zero, when we stopped showing the model how to play and let it discover for itself through exploration, that it became better than any player, and better than the original AlphaGo. A PRM learned from human annotations is AlphaGo. What we would want is AlphaGo Zero.

Which tells us what to aim for. Not a judge but an oracle. And a setting where the two models co-learn: as the generator approaches the reward model’s capability, that progress should unlock new paths of exploration, which the reward model can then verify by itself, without human annotation, as being sound. The reward model could even learn to call on the oracle at inference time, to check intermediate steps as far as the oracle is able to check them.

And there is the question that costs. In Go, something was trivially available that is not available here at all: the simulation environment came with the game. The rules were the oracle. For general reasoning, what is the environment in which an intelligence learns by observing the effects of its own acts? I do not have an answer. I am not sure anyone does.

What is missing is not a model, it is a world

Let me end on the thing I find most interesting in all of this, which is also the least intuitive.

We do not need a machine with human intelligence in order to solve problems a human has not solved. A model that is less intelligent than the best of us, but that tests an enormous number of solutions without tiring, could converge on answers no brilliant human ever found, for the simple reason that no human was going to spend that much time looking.

But that only works in one direction, and the asymmetry matters. Time spent thinking is worth exactly as much as what it can be checked against. Without something that answers back, a model exploring for a month is a model elaborating for a month, and there is no reason for what it produces at the end to be truer than what it produced in the first hour.

An oracle is reality refusing to agree. The whole difficulty of the next step, as far as I can tell, is that we know how to build the thinker and we do not know how to build the thing it thinks against.