The Critic’s Narrowed World: What ChatGPT Actually Learned to Act Upon

Taking the word strictly, ChatGPT is not an innovation. The architecture already existed: it is a Transformer, and Transformers are more than five years old. The training procedure existed too, since PPO, the reinforcement learning method used at the end, was published by OpenAI in 2017. On the other hand, and it is a big one, it is a revolution.

Which leaves a question open: why now. The same mechanisms were available two or three years ago, and we did not get this. The usual answer is that we went and retrieved knowledge already sleeping in the parameters, what Hinton called dark knowledge back when he was working on distillation: information genuinely present in the model, but that we had no access to. I think that is true, and I mostly think it says very little until you look at the means by which we got to it.

That means sits in the third stage of the training. It is the stage people discuss least, and it is the one that decides everything.

A body, and a world to act upon

There is a concept in artificial intelligence called embodiment, which says that intelligence can only emerge once you give an agent a body. A physical body or a virtual one, it does not matter. What matters is what a body implies: it sits in an environment, it can act on it, its action changes that environment, and that change is usable information for deciding the next action. You go round in a loop, with an objective somewhere, if only survival.

That is the reinforcement learning paradigm, and there is nothing new about it either.

There is a distinction to make inside that paradigm, though, and it carries the rest of this article. There are agents that simply learn which action maximises the reward. And there are agents that build themselves a model of the world, and learn to predict the effect of their actions inside that model before taking them.

The difference is easy to feel. If I close my eyes in the room I am in, I can imagine what a given movement would produce without having to make it. I have a room in my head. An agent that only seeks to maximise a reward has nothing of the sort: it has a policy, it does not have a room.

And in a sense, once you have a real model of the world, the reward becomes almost incidental. Once you have understood how that world works, maximising a reward inside it is the easy problem. It is the other half that is hard.

What the critic actually models

Training ChatGPT comes down to three stages.

The first is self-supervised learning: you take a very large volume of text and ask the model to predict what comes next, which teaches it to produce statistically plausible text and nothing else. That alone is enough reason for it to have no mechanism whatsoever for checking what it asserts.

The second produces a second model. Human annotators are given several answers to the same request and rank them, from best to worst. A model is then trained to predict that ranking. This is what we call the reward model, and in reinforcement learning we usually call this kind of model the critic.

The third takes the first model back and turns it into an agent. Its action is to generate text, its environment is the conversation, and it is optimised by reinforcement so that it produces answers the critic scores well.

And this is where I think something gets missed. We usually describe that critic as a filter, or as a guardrail bolted onto the output. In reality, from the agent’s point of view, that critic is its world. It is the only thing it acts on, the only thing that answers it, the only thing it learns anything from. The model of the world I mentioned above is right there: for ChatGPT, the world is the critic.

Except that this world is tiny. It does not model what happens between two people talking to each other. It models a preference ordering over possible answers, as expressed by a group of annotators at a given moment. That is an extraordinarily thin slice of reality, and it is nonetheless the totality of what the agent is allowed to know of it.

So that model of the world is necessarily suboptimal. It was built around the reward instead of being built around the interaction, and those are two different objects.

The price of that narrowing

This narrowing gets paid for, in three places.

The first is political, and fairly obvious once stated. Private institutions are choosing on our behalf what is correct to say, how a conversation should go, which subjects deserve avoiding. They are doing it for a system that will potentially be used by millions of people across thousands of sectors, and they are doing it without anyone having asked them to.

The second is human. It takes an enormous number of people to produce those judgements, and we were not talking much about those people until this week. TIME published an investigation five days ago into the annotators who worked for OpenAI, recruited through an outsourcing firm in Kenya, around thirty people paid between $1.32 and $2 an hour to label text describing the worst of the internet. That particular effort went into training a toxicity detector rather than a reward model, they are two distinct annotation jobs, but the principle is the same in both cases: the acceptable that gets installed in the machine was written by hand.

The third is a limitation of method, and it is the one that interests me most. An agent whose world is a preference model can become extremely good at satisfying a preference. That makes it better at nothing else. The range of answers available to it is bounded by what the critic knew how to score, and there is no mechanism in this setup that lets it step outside that bound.

Decentralising the reward

You could imagine something else, and I find the exercise useful even though it stays hypothetical.

You could imagine the reward coming not from a group of annotators recruited by a company, but from the people actually using the system. Each of them would say what seems good or bad to say, and the signal would come up from there, in a distributed way.

Which would potentially give something other than a single assistant. An agent deployed in a community of gamers and an agent deployed in a scientific environment would not converge on the same place: they would be modelling different interlocutors, therefore different worlds, and they would end up with fairly distant behaviours. One strain at the start, distinct agents at the finish.

The precedent exists, incidentally. When DeepMind trained AlphaStar to play StarCraft, what they trained was a league rather than an agent: several families of agents, some of which had no objective of winning but of exposing the weaknesses of the others. The diversity was built in on purpose, because a single agent optimising a single objective ends up collapsing onto a single strategy.

I am not saying OpenAI should have done that. Constraining the system around a single reward was probably necessary for it to act in a coherent direction, and it was a defensible call. What I am saying, at any rate, is that this call is an architectural choice rather than a fatality, and that it has a cost nobody is measuring today.

That leaves a question I have no answer to, and I am not sure it has one as things stand. If the critic is the agent’s world, then widening that agent’s intelligence amounts to widening its world, and I cannot quite see what a genuine environment for a language model would look like. A conversation with humans, potentially, provided it were treated as an environment to be modelled rather than as an exam to be passed. That would at least be a room, instead of a judge.