“Universal Language Model Fine-tuning for Text Classification” was submitted to arXiv in January 2018 by Jeremy Howard of fast.ai and Sebastian Ruder. ULMFiT proposed a transfer-learning recipe for natural language processing: pre-train a language model on a large general corpus once, then fine-tune it for a specific text-classification task.
The motivation was a gap between fields. Computer vision had long benefited from transfer learning - take a model trained on ImageNet and adapt it - but NLP models were still largely trained from scratch per task. ULMFiT showed the same idea works for language, introducing fine-tuning techniques such as discriminative learning rates and gradual unfreezing to adapt the pre-trained model without destroying what it had learned.
The results were strong: ULMFiT reduced error by 18 to 24 percent across six text-classification datasets, and matched models trained on a hundred times more data using just a hundred labeled examples. Alongside ELMo and the Transformer-based models that followed, ULMFiT helped establish the pre-train-then-fine-tune paradigm that now underpins essentially every large language model.