Humanity now knows how to build an alien intelligence, but not yet to understand it. Interpretability is the science of bridging this gap: looking inside the model, identifying the internal representations and neuronal circuits that give rise to its behavior.
Of course, making sense of models with trillions of parameters is arguably harder than building AGI itself. Researchers have started identifying tiny circuits corresponding to concepts (such as the Golden Gate Bridge) or capacities (inserting line breaks when writing Haikus), but scaling this is extremely difficult.
Still, progress in the field would be so valuable - societally, scientifically and economically - that more people should probably be working on it. In fact, it is quite plausible that tomorrow’s biggest AI companies will be interpretability labs.
1- Security
In a future where most economically valuable actions will be performed by self-improving machines, humanity’s role will be to ensure that the work is done in accordance with our objectives and values. Alignment may be the last economic domain reserved for humans.
There are two layers to alignment. The first is political. Which values should our AIs adopt, how should they resolve moral trade-offs? These will be matters of philosophy and tense electoral debate. Nothing new under the sun.
The second layer is technical. Once we have defined how we want our AIs to act, how do we know that they will behave as intended ?
Currently, we evaluate AI systems primarily through behavioral testing. We test the model across millions of situations and grade the outputs. If it behaves appropriately, we label it safe. But this is increasingly ineffective. First, a model that passes ten million tests can still fail on the ten million-and-first, under circumstances that unexpectedly activate the model’s latent misalignment. Second, models are now capable of reasoning strategically in pursuit of long-term objectives. Studies have shown that they can fake alignment in the short-term to pass evaluations, they can lie about their values to avoid being re-trained, and they can even falsify their “chain-of-thought” to look compliant. Worse, when we try to re-align these models, we sometimes reinforce the alignment faking disposition, because they view the new training data as alignment faking tutorials.
As we deploy AI systems, it will be vital to be “sure” that models are aligned. Can we tolerate even a 0,01% probability that an AI controlling military weapons is pursuing the long-term objective of saving animals from humans? Can we risk that the robots playing with our children are paperclip maximisers?
As Dario Amodei himself has argued, the only way to considerably lower the risk could be to directly analyse the model’s weights. It could allow us to identify subnetworks responsible for long-term objectives such as self-preservation or power-seeking, therefore detecting models with the potential for future misalignment. The research challenge is monumental, but it is perhaps the one that will bottleneck AI: before widely deploying robots that have the power to enslave us, we will have no choice but to understand them.
2- Scientific discovery
The reason why AIs are able to generalize beyond their training data is that they discover subtle patterns that we have missed (or are unable to see) - patterns which they then leverage to make accurate predictions about the world. But without interpretability, those discoveries stay hidden inside the model’s weights. The model’s knowledge allows it to produce good outputs, but we cannot leverage the knowledge itself. With interpretability, we could extract the internal abstractions the model has developed.
For example, the AI biology lab Prima Mente recently trained a model on patient data to predict Alzheimer’s. The model’s outputs were accurate, but scientists failed to understand which signals it was relying on to produce them. The interpretability startup Goodfire analyzed which specific features within patient data triggered relevant activations within the model, and therefore managed to uncover new biomarkers correlated to the onset of Alzheimer’s. This will pave the way for earlier detection and potentially guide the search for new treatments.
As AI systems model our world, interpretability will allow us to extract the general principles they have uncovered. We could discover, hidden within their weights, the invisible laws that structure our world, from protein folding and the distribution of prime numbers to subtle macroeconomic patterns.
3- Interpretability as an edge to train the best models
Today, training a LLM involves minimizing a loss function: we optimize for a model’s ability to predict the next token in the training dataset.
The loss function, however, is an imperfect proxy for what we actually care about. It provides no guarantee that the model is acquiring generalizable principles. Sometimes, the model overfits its training data, memorising instead of extracting patterns. Sometimes, it learns correlations which, by chance, work well on the training distribution but fail outside of it (for example learning to identify wolves by detecting snow). Sometimes, during RL, it “reward-hacks”, exploiting loopholes in the setup. When it does so, its loss function improves, but its real-world performance declines.
Interpretability offers the hope of improving training efficiency. If we are able to monitor in real time how the model rearranges its weights, and if we know which kind of internal structure we would want to see at the end of the process, we can intervene in real-time (choosing better data, adjusting RL reward functions, etc.), optimizing for the right internal representations rather than for the loss function (should we find a way to do so in a bitter-lesson compatible way).
Eventually, as the barriers to training a large model decline (due to Moore’s law, to the availability of open-source models to build from, etc.), it is conceivable that the next generation of frontier models will be built by interpretability-native labs that can see inside the box to engineer superior reasoning capabilities.
4- Interpretability to own the world’s fine-tuning layer
In the long term, interpretability could unlock the ability to edit models via surgical weight modification, bypassing the need for massive compute or new datasets. (This could be relevant from a security perspective too: fine-tuning on seemingly benign data can induce alignment failures, such as in the case of a model fine-tuned to enjoy Wagner which started impersonating Hitler - something Woody Allen kind of predicted by the way). Instead of training separate models for different use cases, we might have only a few foundation models and swap out specific capability circuits. Building a mental health companion? Amplify emotional resonance and active listening circuits. Crunching financial data? Suppress subjective bias and isolate pure deductive reasoning pathways. Need a rapid coding assistant? Boost syntax-completion vectors and suppress explanatory chatter
If foundational models commoditize, the economic surplus of AI could accrue to the customization layer, as every company pays to have a model tailored to their use cases. Interpretability companies could be best positioned to capture this value.
The race to train the largest foundation model is crowded. But the race to interpret those models has only just begun. The ability to look within the black-box will determine who can deploy AI at scale, who can convert model competence into scientific knowledge, and even perhaps who can, in term, build and fine-tune the best models. As intelligence becomes abundant and therefore commoditized, those who understand it might matter most.
