О курсе
Modern LLMs is an engineering course on the complete lifecycle of a language model: from data, tokenization, and architecture to post-training, efficient inference, evaluation, and applied systems.
The course is not designed as another list of model names. Its goal is to provide a system for reading model cards and technical reports, checking architectural arithmetic, estimating the cost of a design choice, and separating durable mechanisms from dated market snapshots.
The course opens with the modern LLM landscape, tokenization, and embeddings. Module 3 develops the basic Transformer step by step, followed by positional encodings, attention, normalization and activations, MoE, and SSMs/Mamba. The route then covers data and pretraining, KV caches, efficient inference and quantization, diffusion LLMs, SFT, preference optimization, reasoning and test-time compute, multimodality, RAG, agents, and evaluation. Module 20 assembles the mathematics of an autoregressive LLM into one construction, from tokens to training and generation, and explains what changes for SSMs and diffusion models.
Formulas are tied to executable checks. The notebooks demonstrate mechanisms, numerical stability, batching, control cases, and reproducibility. They are careful educational implementations, not claims of production readiness: specialized GPU kernels, distributed training, and industrial serving require separate optimization for a particular model, runtime, and hardware stack.
The Stepik edition includes a structured path, auto-graded questions, analytical and computational exercises, detailed solutions, tested English notebooks, separate demonstrations for project exercises, and a final Capstone. In the Capstone, you build a compact MiniLLM from data and byte-level BPE through post-training, inference, and controlled ablations. The project is not graded externally; you preserve and assess the result yourself.
The English and Russian courses are complete, synchronized editions rather than a mixed-language notebook set.
Modern LLMs is part of an evolving series of standalone courses on ML and LLMs. The current series also includes Information Theory for ML: From NLP and LLMs to CV and RL and Reinforcement Learning for LLM. The series may grow to include additional courses; each course is designed to stand on its own, and cross-references offer optional depth rather than required dependencies.
Для кого этот курс
Начальные требования
Familiarity with introductory machine learning is expected: train/validation/test splits, loss functions, gradient descent, overfitting, and the basic probabilistic interpretation of a model.
You should know Python and be able to read PyTorch code. Comfort with tensors, shapes, broadcasting, and simple training loops is sufficient.
Basic linear algebra and calculus are needed: vectors, matrices, inner products, derivatives, gradients, and softmax. Probability is used at the level of distributions, conditional probability, and expectation.
Prior familiarity with the Transformer is helpful but not mandatory. Module 3 develops the basic encoder–decoder architecture, attention, and the main equations in detail. It follows the introduction and Module 2 on tokenization and embeddings.
Jupyter or Google Colab is needed for the practical work. Core demonstrations and smoke routes are designed for accessible environments; a GPU is useful for extended runs but is not required for the theory or most controlled experiments.
Experience with CUDA, distributed training, production serving, or reinforcement learning is not required. The course marks the boundaries of these specialized disciplines explicitly.
Как проходит обучение
The course is self-paced and primarily text-based, with equations, illustrations, tables, and links to primary sources.
The core route contains 17 modules. Optional branch 7b covers diffusion LLMs, Module 18 is a reference on the basic Transformer, and the final module contains the Capstone.
Most modules follow a recurring learning route:
• an engineering question and a precise problem statement;
• the mechanism, equations, and numerical checks;
• modern implementations and limits of applicability;
• auto-graded assessments after the main sections;
• analytical, computational, and project exercises;
• detailed solutions, a notebook, and key papers for further study.
Assessments are graded automatically. Regular exercises are not an external examination: attempt a problem first, then open the detailed solution. Project exercises form a separate practical route, with demonstration notebooks available for selected projects.
The English and Russian courses use separate notebooks. They run in Colab or locally and focus on testable mechanisms and small experiments; extended profiles may require a GPU.
The Capstone combines pretraining, post-training, inference, and ablations in a MiniLLM project. It is not reviewed externally: you retain the config, seed, metrics, final weights or a checkpoint, write a report, and use the self-assessment table.
A study load of roughly five to seven hours per week is recommended, but there is no fixed schedule. The reference module and optional advanced branches can be read selectively.
Программа курса
В курс входят
- 414 уроков
- 415 тестов
Наши преподаватели
Отзывы прошедших курс
Что вы получаете
- 20 main modules and a MiniLLM capstone: 414 lessons containing 945 text and assessment steps.
- 415 auto-graded questions placed after the relevant explanations and practical work; explanations appear after a correct answer.
- 116 analytical, computational, and project exercises with detailed expandable solutions, plus self-check questions.
- 42 notebooks: 20 main notebooks, 18 project demonstrations, and 4 capstone notebooks. Individual downloads and a complete course bundle are provided.
- 180 illustrations covering mechanisms, quantitative comparisons, and numerical examples.
- A complete educational MiniLLM codebase covering the data pipeline, tokenizer, dense/MoE decoder, training and resume, post-training, generation, caches, speculative decoding, fake quantization, probes, and reporting.
- Scalable MiniLLM profiles: dense S, M, and L with MoE twins—from the 6.3M-parameter S model to dense L at 1.02B and L-MoE at roughly 1.06B active parameters per token (10.02B total across all experts); large profiles act as verifiable compute/memory passports and do not require full training.
- The MiniLLM codebase comes with a comprehensive validation suite covering architecture, training, data, post-training, and inference.
- Curated key papers and primary sources, engineering tables, and practice measuring memory, latency, parameter counts, loss, and reproducibility instead of relying on a single headline metric.