The Needle In A Haystack test was created by Greg Kamradt in November 2023 as a quick way to pressure-test the long-context claims that model providers were starting to make. The idea is deliberately simple: take a long body of text (the haystack), plant a single unrelated fact somewhere inside it (the needle), and then ask the model to retrieve that fact. By sweeping the needle across many positions and the haystack across many lengths, the test produces a grid showing where retrieval succeeds and where it fails.
The early runs became widely shared because they revealed uneven behavior. GPT-4 with a 128,000-token window, tested in November 2023, showed degraded retrieval when the fact sat in the middle of the document rather than near the ends, while Claude 2.1 showed more consistent retrieval across positions. The visual heatmaps made an abstract limitation concrete and easy to communicate, and the open-source code let anyone reproduce the test on new models, which is why so many later benchmarks and provider announcements adopted the format.
Its importance is twofold. It gave practitioners an accessible reality check on long-context marketing, and it exposed that retrieval can depend on where information sits, not just how much fits in the window. Later benchmarks like RULER were built specifically because this single-needle test, while influential, was too easy on its own.