AlexNet was trained on two consumer GPUs

The neural network that reignited deep learning, AlexNet, was not trained on exotic supercomputers. Its authors, Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, trained it on two NVIDIA GTX 580 GPUs, the same class of graphics card sold to PC gamers. In the paper “ImageNet Classification with Deep Convolutional Neural Networks,” they wrote that “to make training faster, we used non-saturating neurons and a very efficient GPU implementation of convolutional nets,” for a network with 60 million parameters that would have been impractical to train on the CPUs of the time.