“AutoML-Zero: Evolving Machine Learning Algorithms From Scratch” was submitted in March 2020 by Esteban Real, Chen Liang, David R. So, and Quoc V. Le of Google. It pushed the idea of automated machine learning to an extreme: rather than searching over hyperparameters or neural network architectures built from human-designed components, it tried to evolve entire learning algorithms using only basic mathematical operations as building blocks.
The system represents a candidate algorithm as three small programs (setup, predict, and learn) made of primitive operations like addition, multiplication, and matrix operations, and then uses evolutionary search to improve them. Starting from empty programs, the search rediscovered classic techniques on its own, including two-layer neural networks trained with backpropagation, and it went on to evolve useful tricks such as normalized gradients and weight averaging when tuned for specific tasks like CIFAR-10.
The work was a striking demonstration that fundamental ideas of machine learning can emerge from search rather than human insight, though at large computational cost and on small problems.
For a business reader, AutoML-Zero is a thought-provoking marker of how far automation of the research process itself might eventually go.