“Inverse Scaling: When Bigger Isn’t Better” was submitted to arXiv on June 15, 2023 by Ian R. McKenzie, Ethan Perez, Samuel R. Bowman, and a large group of collaborators. It assembled hard evidence against the convenient belief that a language model’s performance reliably improves as it gets larger.
To find counterexamples systematically, the authors ran a public competition, the Inverse Scaling Prize, inviting researchers to submit tasks on which bigger models do measurably worse. The result was eleven datasets exhibiting genuine inverse scaling. Analyzing them, the authors identified four recurring causes: models preferring to repeat a memorized sequence over following the actual instruction; models faithfully imitating undesirable patterns present in their training data; models latching onto an easy distractor version of a task instead of the intended harder one; and misleading few-shot examples that lead larger models further astray.
The broader lesson is about prediction. Smooth scaling laws had encouraged the assumption that you can extrapolate a small model’s behavior to forecast a large one’s. Inverse scaling shows that for some capabilities this fails, and that data quality and the precise training objective deserve as much attention as raw size. It is a natural counterpart to work on emergent abilities, which documents capabilities that appear only at scale.
For a general reader, the paper is a useful corrective to scale-at-all-costs hype: sometimes a bigger model is not a better one, and knowing where that happens is essential for deploying these systems safely.