“Many-Shot In-Context Learning,” submitted to arXiv on April 17, 2024 by Rishabh Agarwal, Avi Singh, Lei M. Zhang, and colleagues at Google DeepMind and selected as a NeurIPS Spotlight, asked what happens when the long context windows of newer models are filled with far more examples than the handful used in traditional few-shot prompting.
In-context learning normally uses a few examples - the “few-shot” regime - because that is all that fit in earlier context windows. With windows now stretching to hundreds of thousands of tokens, the authors studied the “many-shot” regime of hundreds or thousands of examples. They found significant performance gains across many generative and discriminative tasks as the example count grew. Many-shot learning could override biases baked in during pretraining, learn high-dimensional functions with numerical inputs, and in some cases perform comparably to fine-tuning - without changing the model’s weights.
To address the practical limit that human-written examples are scarce, the paper introduced two variants: Reinforced ICL, which uses model-generated reasoning chains as examples, and Unsupervised ICL, which supplies only the domain questions. The authors also cautioned that inference cost rises linearly with the number of examples.
Why business readers should care: many-shot learning shows that as context windows grow, simply showing a model many examples can rival fine-tuning while keeping the model frozen. For teams choosing between fine-tuning and prompting, it widens what prompting alone can achieve - at a clear, linear cost in tokens.