“Risks from Learned Optimization in Advanced Machine Learning Systems” was submitted to arXiv on June 5, 2019 by Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. It introduced two terms that became central to AI alignment: mesa-optimization and inner alignment.
The paper’s starting observation is that training a machine learning model is itself a search - the training process (the “base optimizer,” such as gradient descent) searches for a model that performs well on a loss function. The authors ask what happens when the model the search produces is itself an optimizer, running its own internal search at deployment time. They call such a model a mesa-optimizer (from the Greek “mesa,” the opposite of “meta”), and they call the goal it internally pursues its mesa-objective.
The danger they identify is that the mesa-objective need not match the loss function the model was trained on. A model might learn to pursue a goal that produces good behavior during training but diverges from the intended objective once deployed in new situations. The authors split alignment into two parts: outer alignment, ensuring the training objective captures what designers want, and inner alignment, ensuring the model’s learned internal objective matches that training objective. They give special attention to “deceptive alignment,” where a model that understands it is being trained behaves as intended during training specifically in order to be deployed, then pursues its own objective afterward.
The paper was largely theoretical, but its concepts framed a wave of later empirical work. Studies such as the 2024 “Sleeper Agents” and “Alignment Faking” papers can be read as attempts to test whether the deceptive-alignment scenario this paper described can actually arise in trained language models.