Article 4 1990-2011

The Quiet Proving Years

LSTM, Deep Blue, reading checks at scale, and the dataset that changed everything.

Choose how you'd like to follow along

On the 11th of May, 1997, in a building in midtown Manhattan, the reigning world chess champion - a man many consider the greatest who ever lived - put his head in his hands and resigned. Garry Kasparov had just lost a match to a machine. An IBM computer called Deep Blue had beaten the best chess player on the planet, and the photograph of Kasparov, slumped and stricken, went around the world. The machines, everyone agreed, had finally arrived.

Except they hadn’t. Not really. And this is the great misdirection at the heart of our story. Because Deep Blue was not, in any meaningful sense, thinking. It was not learning. It had no idea it was playing chess. It was a magnificent piece of brute-force engineering that could calculate two hundred million board positions a second, and it won the way a bulldozer wins an arm-wrestling match. The famous moment that the whole world watched - the moment everyone remembers as the dawn of AI - was a kind of dead end. The real revolution was happening somewhere else entirely, somewhere nobody was pointing a camera, and it would take another fifteen years to arrive.

This is the chapter nobody tells you about. The quiet years, from 1990 to 2011. No grand promises, no famous winters, no slumped champions. Just two decades of patient, invisible work, during which the world quietly assembled every single ingredient that the modern AI explosion would need - and almost nobody noticed it was happening.

Here is what was really going on. For most of these years, neural networks - the brain-inspired approach we have been following - were still out in the cold. The fashionable methods were built on statistics. A technique called the support vector machine, developed by Vladimir Vapnik in 1995, was elegant, mathematically gorgeous, and it beat neural networks on most problems people cared about. If you were a smart young researcher in 1999 and you wanted a job, you did not work on neural networks. You would have been laughed out of the room.

But the embers never quite went out. In 1997, two researchers named Sepp Hochreiter and Jurgen Schmidhuber solved a nagging problem that had crippled networks dealing with sequences - the tendency to forget what happened more than a few steps ago. Their fix, called Long Short-Term Memory, gave networks a kind of working memory, and it would one day power the first wave of usable speech recognition and translation. The idea survived. It was just waiting.

And in the background, three things were piling up. Three ingredients.

The first was raw computing power, and it arrived from the most unlikely place imaginable: video games. A company called NVIDIA made graphics chips - GPUs - to render explosions and monsters for teenagers. And it turns out that the math you need to draw a video game world is almost exactly the math you need to run a neural network: thousands of simple calculations, all at once, in parallel. Nobody designed it this way. The engine of the artificial intelligence revolution was built, essentially by accident, so that kids could play better video games. In 2007, NVIDIA opened up its chips for general use, and the most powerful tool the AI pioneers could have asked for was suddenly sitting on store shelves.

The second ingredient was data, and it came from the world moving online. Every photo uploaded, every page written, every click - an ocean of information, growing every day. And people built clever ways to harness it. Amazon created a marketplace where you could pay people a few cents to label images. Netflix offered a million-dollar prize to anyone who could improve its movie recommendations, and turned machine learning into a competitive sport. Out in the desert, the Pentagon ran a race for self-driving cars - and in 2004, not a single car finished; the best one managed about seven miles before it gave up. One year later, in 2005, a Stanford team led by Sebastian Thrun built a car that drove a hundred and thirty miles across the desert and won. The pieces of autonomy were coming together.

And the third ingredient - the spark that would set the other two alight - came from a Stanford professor named Fei-Fei Li, who had what sounded, at the time, like a slightly crazy idea. Everyone in the field was obsessed with building cleverer algorithms. Li thought they had it backwards. What if the algorithms were fine, and the real problem was that they had never been fed enough? What if you gave a learning machine not a few thousand examples, but millions? So starting in 2009, she built something called ImageNet - a database of over a million images, each one labeled by hand, by humans, across thousands of categories. It was a staggering, almost absurd amount of work. And then she did the most important thing: she turned it into an annual contest. Whoever could build the system that best recognized those images would win. The gauntlet was thrown.

And waiting in the wings were the believers. Remember Geoffrey Hinton, our stubborn neural-network evangelist from the last chapter? Through these lean years, a Canadian research institute had quietly kept him funded, along with two others who had never given up - Yann LeCun, the zip-code reader, and Yoshua Bengio. They kept the small, unfashionable community of neural-network researchers alive and together, through the entire winter, waiting.

So by 2011, look at what was sitting on the table. The algorithm - backpropagation - had existed since the 1980s. The computing power had arrived, by accident, from gaming. The data had arrived, in a flood, from the web. The believers had been kept alive, funded, and ready. And a contest stood waiting, with a giant pile of labeled images, to reward whoever could finally put it all together.

For twenty years it had felt like nothing was happening. Like AI was a backwater, a field of broken promises. But that quiet was an illusion. It was not a dead field. It was a fuse, fully laid, stretching across two decades, waiting for a single spark.

In 2012, at that ImageNet contest, somebody finally struck the match. And the quiet would end all at once.

Listen to the article

The full article, read aloud. Runtime 7:11.

The video episode is coming soon

We're producing narrated video editions on YouTube - the same story, walking the timeline with visuals. In the meantime, read the article or listen to it.

Sources and show notes

Every claim in this article is drawn from the AI Library, where each entry links to its own primary source - the original paper, the official record, the actual document.

IBM reframes translation as statistics, not rules IBM reframes translation as statistics, not grammar rules. TD-Gammon teaches itself backgammon TD-Gammon teaches itself backgammon by self-play. Support-vector networks: the method that out-ran neural nets Support-vector networks - the method that out-ran neural nets. Support Vector Machine The wide-margin classifier that dominated 1990s-2000s ML. Vladimir Vapnik Co-creator of the support vector machine. Long Short-Term Memory networks introduced Long Short-Term Memory solves the vanishing-gradient problem. Long Short-Term Memory (LSTM) A recurrent network that remembers across long gaps. Sepp Hochreiter Co-inventor of the LSTM. Juergen Schmidhuber Co-developed the LSTM. Deep Blue defeats world chess champion Kasparov Deep Blue beats world chess champion Garry Kasparov. LeNet-5 and gradient-based document recognition LeNet-5 reads handwritten documents end to end. Gradient-Based Learning Applied to Document Recognition (LeNet-5) The LeNet-5 paper, gradient-based document recognition. Judea Pearl Invented Bayesian networks and the mathematics of causality. Bayesian Networks Graph-based reasoning about uncertainty. NVIDIA is founded to build 3D graphics chips NVIDIA is founded - the company that would power deep learning. NVIDIA ships the GeForce 256, the first chip called a GPU The GeForce 256, the first chip called a GPU. NVIDIA releases CUDA CUDA lets general-purpose code run on GPUs. iRobot launches the Roomba and autonomy enters the home The Roomba puts autonomy in millions of homes. Stanley wins the DARPA Grand Challenge Stanley wins the DARPA Grand Challenge. The 2004 DARPA Grand Challenge that nobody finished The 2004 race that nobody finished. Sebastian Thrun Led the winning Stanford self-driving team. Boss wins the DARPA Urban Challenge Boss drives itself through mock city traffic. Amazon launches Mechanical Turk, a marketplace for human microtasks Mechanical Turk, a marketplace for human labeling. Netflix offers $1 million to beat its recommender The Netflix Prize mainstreams the public ML competition. ImageNet, a large-scale image database ImageNet, the labeled image database that changed everything. Fei-Fei Li Created ImageNet. ImageNet Challenge (ILSVRC) The annual competition whose 2012 result lit the fuse. The unreasonable effectiveness of data The 2009 case that data beats cleverness. Deep belief nets relaunch deep learning Hinton's deep belief nets relaunch deep learning. CIFAR Funds Deep Learning Through the Winter (2004) CIFAR funds Hinton, Bengio, and LeCun through the winter. IBM Watson wins Jeopardy! IBM Watson wins Jeopardy! Apple puts a talking assistant in millions of pockets Apple puts a talking assistant in millions of pockets. Google reveals its self-driving car project Google reveals its self-driving car project. Geoffrey Hinton Kept neural networks alive through the lean years.

Made with AI, sourced like a library

We are open about this: this article is produced with AI. The research, the writing, the narration - AI does the heavy lifting, with human review before anything is published. That is not a footnote; it is the point. CenterConsulting builds software with AI every day, and this series is the same method applied to storytelling.

What keeps it honest is the foundation: every article is built on the AI Library, our knowledge base where every fact links back to a primary source. The show notes above list this article's entries, so you can check everything yourself.