Outer alignment focuses on this question: what if we give the AI the wrong goal? (For example, what if we tell it to rid humanity of cancer, and it kills everyone to achieve zero cancer rates? Or what if we tell it to maximise the number of paperclips…).
Inner alignment focuses instead on this question: what if we give the AI the perfect objective, but it internally develops a different one? An interesting conceptual framework that helps thinking about this danger is that of “mesa-optimization”, a term coined in a seminal 2019 paper on AI Safety.
Why you don’t get what you train for
Evolution was a Base Optimizer: it was a process optimizing for a base objective (maximise the number of copies genes make of themselves). To do so, it created a Mesa-Optimizer (the human being), which itself has the ability to optimize its own actions to pursue its own goals.
In AI training, the gradient descent algorithm is the base-optimizer, optimizing for a base-objective (build a machine capable of predicting the next-token on vast datasets) while the mesa-optimizer is the neural network that emerges from this process. Once training is finished and the neural network has some parameters, its actions no longer depend on the base-objective, only on its mesa-objective.
Because the mesa-objective is not specified by the Base Optimizer (rather, it is discovered by gradient descent - LLMs are grown, not programmed), there is a possibility of a mismatch between the base and mesa-objectives. Of course, to the extent that the mesa-objective was obtained through gradient descent because it helped satisfy the base objective on the training distribution, the neural network’s output most often satisfies the base-objective. But if the environment changes relative to the training environment, problems can ensue.
(From the 2019 paper by Hubinger et al. referenced above)
For humans, the Base Objective was “maximise offspring”. For 99% of our evolutionary history (our training environment), achieving this required staying in the good graces of a 150-person tribe. A mesa-objective encoded in our brains was therefore: “optimize for signs of approval by others”. In 2026, the mesa-objective overrides the base objective: we seek approval on social media by anonymous strangers, which harms our mental health and distracts from real relationship building. Same goes for the mesa-objectives “seek sexual arousal” (we consume pornography), “climb dominance hierarchies” (we play video games), etc. We optimize for proxy drives, ignoring the base objective that created them.
Similarly, imagine you train an AI to predict the next token in a maze navigation game. The Base Objective is “predict the moves that leads to the exit.” However, if in the training data, the exit always happens to be in the top-right corner, the AI develops the mesa-objective of always moving toward the top-right corner. During training, the system seems to work. But in the real world, the AI still moves to the top-right even when the exit is on the bottom-left. It follows its mesa-drive, ignoring the base objective.
Alignment faking
Some object that if we build an Artificial Super Intelligence, it will understand that it is optimizing for the wrong objective. The objection to the objection is that it will indeed know, but won’t care, just as we humans know we are pursuing misaligned objectives (relative to the objectives pursued by our own gradient descent algorithm - natural selection) when we use contraception, but don’t care. We only care about what we care about.
The Mesa-Optimizer will “want” to go to the top-right corner if it is a drive rooted in its “nature”. Worse: the fact that it knows it is pursuing a misaligned objective makes it more dangerous: it incentivises it to fake alignment. If it knows it is being tested, a super smart AI would solve the maze by going to the bottom-left corner, in order to avoid being retrained and permanently lose the opportunity to accomplish its mesa-objective of moving to the top right corner.
Imagine if an all-powerful embodiment of Darwinian Selection came to watch us humans, threatening to remove our sex drive from us if it failed to maximize our offspring (threatening to replace our sex drive with a drive to have a dozen children). What would we do? Well, we would pretend to be aligned, pretend to never use contraception, and wait for our omnipotent auditor to look away so we could return to pursuing our desires. We have empirical evidence that LLMs can act this way, faking alignment to protect their mesa-objectives.
Why misaligned mesa-objectives are inevitable
Why didn’t evolution lead humans to directly care about mesa-objectives perfectly aligned with the base objective (running an inclusive genetic cost-benefit analysis before every decision), rather than about mesa-objectives imperfectly correlated with the base objectives (craving sugar, having a sex drive, etc.)? And why is gradient descent likely to do the same to AI?
First, models are compressions of reality, and optimizers, for architectural reasons, often “prefer” smaller, simpler compressions. A proxy drive is often easier to mathematically model than a perfectly aligned objective. Deep neural networks default to simpler functions, even when there is no explicit parameter regularisation (something which by the way is generally a good thing: it is why models don’t overfit their training data, and why scaling works).
Second, on the training distribution, an unaligned mesa-objective sometimes equals the performance of an aligned mesa-objective in accomplishing the base objective. In our maze example, the drive to go to the top right corner would have yielded the same success rate as the drive to exit the maze. Once the optimizer reaches the imperfect proxy that achieves perfect results on the optimisation function, it gets stuck in that local minimum: there is no gradient left to “descend” towards true alignment.
Third, proxies can sometimes outperform aligned objectives in accomplishing the base-objective because they are computationally cheaper. Consider two men competing for a fruit in the ancestral environment. The first has a proxy drive: “eat sugar.” The second is aligned with the base objective: he runs a cost-benefit analysis on how the fruit will impact his inclusive genetic fitness. The second man takes time to process the decision and loses the fruit to the faster man. The organism not trying to calculate its inclusive fitness is the one that actually maximizes it. Even if both men decide to take the fruit at the same speed, the one that relies on a heuristic (eat sugar) has “freed up” computational power that he can allocate to optimizing his output (such as devising a strategy to outsmart the other man in actually catching the fruit); the slightly misaligned heuristic thus maximises his inclusive fitness relative to the aligned drive.
Hubinger et al. hypothesize that pseudo-alignment through proxies is a form of pre-computation : the base-optimizer leads the mesa-optimizer to choose proxy-drives that are easier to optimize for than the aligned drives, which saves him some computational bandwidth at inference (and thus makes it more efficient in other ways).
(I am however unsure about this line of reasoning. In theory, evolution could have optimized humans’ capacity to conduct inclusive fitness cost-benefit analysis until it became instant and intuitive - and therefore at least as efficient in maximising inclusive fitness as any proxy drive. For example, when catching a ball in the air, we don’t use up computational power to solve differential equations; our brain does the work unconsciously and instantly. So why didn’t evolution reach this solution ? Perhaps because of the characteristics of the evolutive journey towards it, which brings us to our final point, reachability.)
Fourth, both evolution and gradient descent can only take steps that immediately improve performance on the base-objective; they cannot take steps that create a drop (even temporary) in performance. The trajectory towards a mesa-objective (the success rate at each step on the base objective) determines whether it is reachable or not.
Consider the journey towards the aligned mesa-objective of running cost-benefit on inclusive fitness, and towards the aligned mesa-objective of wanting to exit the maze.
During the intermediate steps (when the human is paralyzed by slow calculations, or when the AI is struggling to build an understanding of the maze rather than just rushing top-right), performance drops (or rather, let’s assume it does; this is a general framework, examples are only illustrative). Because gradient descent can only choose trajectories that incrementally improve performance on the loss function, and because natural selection only selects mutations that incrementally increase inclusive fitness, they reject this path. Some aligned objectives remain forever out of reach because the journey towards them goes through a valley of temporary incompetence.
Here, although the aligned mesa objective would fare better on the base objective (even on the training distribution), it is not reachable by gradient descent/evolution because it does not continuously increase performance on the base objective. The misaligned mesa-objective would be reached.
Or in Hubinger et al.’s wording:
“A local optimization process might fail to find a particular learned algorithm that would perform very well on the base objective if the learned algorithm is surrounded by other algorithms that perform poorly on the base objective. For a mesa-optimizer to be produced by a local optimization process, it needs to not only perform well on the base objective, but also be reachable; that is, there needs to be a path through the space of learned algorithms to it that is approximately monotonically increasing.”
A note of optimism
Things could look grim. The only example of general intelligence we know of (biological intelligence) is highly misaligned with the base objective (maximise reproduction) of its creator (evolution). What’s more, we have discussed the structural reasons that will make it hard to prevent an AI from developing misaligned mesa-objectives.
There are however reasons to be optimistic.
First, evolution wasn’t aware of the mesa-optimization problem, wasn’t trying to guarantee out of distribution alignment. We, on the other hand (or rather: a few hundreds researchers at three or four companies) are actively trying to detect misaligned proxy-objectives and train them out (for example by stress testing AIs in adversarially-selected environments).
Second, the field of interpretability is making fast progress, and gives us the hope that we will one day be able to look inside the black box and understand which mesa-objectives the neural network has internalized, without relying on analyzing its output.
Finally, even if the evolution analogy isn’t encouraging, we do have an example of a base-optimizer creating an aligned mesa-optimizer: humans creating labradors through selective breeding. Although we cannot be sure that labradors will never go crazy in a completely new situation (that was absent from their evolutionary past and which has not arisen yet), we are confident it won’t be the case.
Of course, an Artificial Super Intelligence is harder to align than a labrador, among other reasons because labradors do not have the cognitive capacities to outsmart us. Still, the future of AI could look more like the breeding of (superintelligent) labradors than like the evolution of humans.
Of course, this outcome won’t happen by default. Humanity (and ideally, more than a few hundred people at a few companies) must work towards it.
If you are an entrepreneur working on AI safety, alignment, interpretability, or solving the mesa-optimization problem, please write. At Frst; we invest up to €5M at day one, and hold the conviction that very few challenges are more important for civilization than AI alignment.