This is Sebastian Raschka’s tutorial “Pretraining and Finetuning LLMs from the Ground Up,” delivered at the SciPy 2024 conference and posted on the official SciPy YouTube channel. Raschka is the author of several widely used machine-learning books and a research engineer who has written extensively on implementing LLMs in PyTorch, and the session runs roughly three hours as a guided coding walkthrough.
He builds a small GPT-style language model from scratch, covering the data input pipeline, the core transformer architecture components, and the pretraining loop, then shows how to load pretrained weights and fine-tune the model using open-source tooling. Rather than treating models as black boxes, he writes the building blocks in code so that the attention mechanism, the training objective, and the fine-tuning steps are all visible and understandable.
This is a deep, practitioner-focused talk for people who learn by reading and writing code. For a technical reader, it removes the mystery from how modern language models are actually constructed and adapted, which is valuable for anyone deciding whether to fine-tune a model rather than rely solely on a hosted API.