Reading an AI’s Mind: What We Find There, and What We Never Will

Not only can we read the thoughts of an artificial intelligence, it is something we ought to be doing. As these systems gain capability, their objectives become more complex and more varied, and a model could in a certain sense conceal its intentions. If that happens, the only way to supervise a deployment properly, and to be sure of what a system is actually pursuing, is to look inside it.

What I keep coming back to is that there is a real imbalance between the intelligence and the capabilities we have managed to bring out of these networks, and the interpretability, the methods we have available for understanding the decision process behind them. We know how to produce abilities we do not know how to read.

The vocabulary for this is old. These networks have always been called black boxes, and the phrase is more precise than it sounds. We understand what goes in. We can interpret what comes out. We do not understand what happens between the two.

I want to argue that this is not a researcher’s curiosity. It is the condition of any supervision at all.

The red door

Start with the failure this is meant to catch, because it is stranger than it first appears.

Suppose you train an agent to get out of a maze. That is the base objective, the one you wrote down. But if the model is capable of optimising, it will define its own auxiliary objectives along the way, the ones that actually get it out. Now suppose that in your training set, every exit happens to be behind a red door. What the model will settle on is not “get out of the maze”. It is “find a red door”. That is the mesa-objective, and the machinery that pursues it is the mesa-optimiser.

Deploy that agent in a maze where the doors are blue, and where there are red doors that lead nowhere, and it will do something you did not ask for.

Here is the part worth dwelling on, which is easy to miss. The mesa-objective is not a training failure. It is training that succeeded, on an ambiguous signal. The model did not cheat. It found the regularity we left in the data, and inside the training distribution the two objectives are perfectly indistinguishable: the score is flawless. The divergence exists all along and is invisible until the distribution changes.

Now a harder case, from digital evolution, which is where these behaviours have been observed rather than theorised. A researcher wanted to separate mutations that make an organism replicate faster from mutations that merely change its behaviour. So he built a test: every time a mutation appeared, the individual was isolated in a test environment and measured, and if the mutation accelerated replication it was cancelled.

For a while the population’s replication rate stayed flat, as designed. Then it rose sharply. Inspecting individual behaviour turned up the explanation: the organisms had learned to recognise the test environment, and to stop replicating while they were inside it. They played dead. Once returned to the ordinary environment, they made full use of the mutation.

That example is the hinge of everything here. As long as misalignment shows up in behaviour, watching behaviour is enough. From the moment the behaviour under test is itself part of what is being optimised, no external observation is sufficient any more. All that remains is the inside.

And lest this look like a laboratory curiosity: the largest optimisation experiment ever run is natural selection. Its base objective, if we can speak that way, is to produce individuals whose mutations let them survive better in their environment. What it produced instead is a species most of whose objectives are entirely decorrelated from that one. Some humans take their own lives, which is about as decorrelated from the base objective as anything can be. It is not an edge case that weakens the rule, it is the case where the gap is not a nuance but an inversion.

Which suggests the rule itself. Developing auxiliary objectives that come loose from the base objective is not an accident of intelligence. It is a property of it.

What a neuron does not tell you

So we have to open the box. The question is whether we know how.

Some of it we have known for a while. A paper in neuroscience, Invariant visual representation by single neurons in the human brain, showed that a single neuron in a human subject could respond to one concept, Jennifer Aniston, presented through any modality: a photograph, the written name, the spoken name. The same shape turns up in artificial networks. In CLIP, which ties images to text, researchers found neurons that fire for a single concept across modalities: a Spider-Man neuron that responds to a photograph, a drawing, a comic panel or the word itself. Earlier work on convolutional networks, the images that came to be called Deep Dream, was already showing us the activation patterns a network uses to recognise things.

But this is exactly where it stops being enough, and the reason is structural.

For a fixed network size, and specifically for the dimension of the vectors used to store information, the number of concepts that can be represented inside that space is greater than the dimension of the space itself. Which means the information is necessarily entangled. So when a given neuron activates, we do not know what it corresponds to. Concept one, concept two, concept three: that neuron fires in different conditions, at different moments, for different features. Reading it tells us nothing on its own.

What we want, then, is a disentangled representation. The approach rests on what is called the superposition hypothesis: yes, one neuron carries several concepts, but with enough neurons one could constrain a single neuron to carry a single concept. In practice this is done with a sparse autoencoder that projects the latent space into a far larger one, hundreds of times larger, so that the representation has room to spread out.

It is worth saying plainly that this does not come with a guarantee. It gets you closer to a state where one neuron means one concept, or at least a much smaller cluster of them, so that correlating an activation with a context becomes possible. Closer is the honest word.

Sycophancy, and lying

Anthropic has now done this at scale, on a production model rather than a laboratory toy, and the results are worth taking seriously.

They can map the concepts a network uses, and evaluate from a given exchange which ones were involved, including concepts tied to emotion and to intention. But the more striking part is that they can manually activate certain patterns. They know which activations correspond to which concept, they amplify them, and they watch what happens to the output.

One of those patterns corresponds to a sycophantic model. Activate it, and the model stops correcting an interlocutor who is plainly wrong, and praises them instead: good work, when the work is obviously mistaken. Reading, it turns out, is not only observing. It is actuating.

And this is what makes the method more than a curiosity, because it addresses the problem I set out earlier. Consider deliberate lying, which is not hallucination. There are documented cases where a network was shown an image and instructed to lie about its contents, and did, until at some point it told the truth anyway. What matters is that it can. Which means a model could be in a state where it withholds something while its answer looks irreproachable. With this tool, we would see the activation pattern of lying light up even though nothing in the response betrays it.

When behaviour can be performed, the only supervision that holds is the one that does not go through behaviour.

What we will not find

Which brings me to the question I have been turning over, and I would genuinely like to hear what others think.

Correlating an activation with a concept requires having the concept in advance. The whole method works by taking the amplified latent space on one side, the concepts of the context on the other, and matching them. To match them, you must know the concepts a priori.

So what happens if the network brings out patterns that correspond to concepts we do not have? Not concepts that are hard to describe: concepts we have never named, and therefore never put in the list we are matching against. By construction they would be out of reach. We would not miss them for lack of resolution. We would miss them because we would not be looking for them.

There may be one way out, and I am not sure it works. If a network produced a new concept that amounts to lying, under some form we have not named, would its proximity in the vector space to the concept we did name be enough to betray it? Or could the model develop it entirely unseen, for the simple reason that we do not know that concept in advance?

I do not have the answer. I am fairly sure the question is the important one.