Чему вы научитесь
- Formalize LLM training and inference problems as an MDP, a contextual bandit, or full-response selection by stating the state, action, reward, and horizon explicitly.
- Compute returns, V/Q/A functions, Bellman targets, Monte Carlo and TD estimates, and reason about the bias–variance trade-off.
- Apply REINFORCE, actor–critic, GAE, and PPO; analyze the policy loss, value loss, clipping, entropy bonus, and KL regularization.
- Distinguish on-policy, off-policy, imitation learning, and offline RL; check support/coverage assumptions and interpret off-policy evaluation.
- Build preference-data and reward-model pipelines; analyze annotation noise, calibration, LLM-judge biases, and reward hacking.
- Compare RLHF, DPO and related preference objectives, RLVR, and GRPO by their data source, feedback contract, and estimator.
- Design verifiers and group-relative signals while separating Best-of-N and test-time search from policy-parameter updates.
- Design agentic RL environments and rollout contracts covering tools, partial observability, stop reasons, masks, policy versions, and stale data.
- Organize asynchronous RL infrastructure and reproducible experiments with versioned checkpoints, data, rewards, configurations, and logs.
- Evaluate LLM/RL systems at matched budgets using pass@k, paired comparisons, confidence intervals, multiple seeds, and an explicit failure taxonomy.
- Run robustness and safety evaluations, detect proxy overoptimization, and write protocol passports that make results auditable.
- Translate formulas into notebook experiments, verify numerical invariants, and explain the limitations and failure modes of the results.
О курсе
Reinforcement Learning for LLM is an engineering course on modern reinforcement learning for language models and agentic systems: from classical RL foundations to post-training, RLVR, search, infrastructure, and safety-aware evaluation.
The course is not organized as a catalogue of algorithms. Its goal is to teach you to choose a method from the data and feedback contract, identify the policy being optimized, validate estimators and masks, understand rollout cost, and distinguish training from inference-time candidate selection.
The program contains 13 modules. The first five provide a self-contained route through MDPs/POMDPs, value functions, bandits, policy gradients, and offline RL. The course then treats an LLM as a token-level policy and covers reward models and preference data, RLHF and DPO, RLVR/GRPO, test-time search, and agentic RL. It concludes with rollout and training infrastructure, statistical evaluation, reward hacking, contamination, and safety.
Equations are tied to executable checks. Small NumPy references, tests, and notebooks help validate objectives, estimators, KL, masks, numerical stability, and reproducibility. These are careful educational implementations, not claims of production readiness: distributed rollouts, specialized GPU kernels, and industrial serving require separate optimization.
The Stepik edition includes a structured path, auto-graded assessments, analytical and computational exercises, detailed solutions, and notebook practice.
The lecture texts and illustrations—without assessments, exercises, solutions, or the practical notebook route—will be published openly on GitHub and possibly on a separate website. The English and Russian courses are complete, synchronized editions.
Reinforcement Learning for LLM is part of an evolving series of standalone courses on ML and LLMs. The current series also includes Information Theory for ML: From NLP and LLMs to CV and RL and Modern LLMs. The series may grow to include additional courses; each course is designed to stand on its own, and cross-references offer optional depth rather than required dependencies.
Для кого этот курс
Начальные требования
Familiarity with introductory machine learning is expected: train/validation/test splits, loss functions, gradient optimization, overfitting, and the basic probabilistic interpretation of a model.
You should know Python and be able to read basic PyTorch code. Comfort with tensors, shapes, broadcasting, and simple training loops is sufficient.
Basic probability, linear algebra, and calculus are needed: distributions, conditional probability, expectation, vectors, matrices, derivatives, and gradients.
It is helpful to understand Transformers, tokenization, autoregressive generation, softmax, and log-probabilities. Modern LLMs offers additional depth but is not a prerequisite.
No previous reinforcement-learning course is required: Modules 1–5 introduce the necessary classical machinery. Readers who already know RL can use them as a fast refresher.
Jupyter or Google Colab is needed for the practical work. Core exercises run on CPU and in accessible environments, while a free Colab T4 covers short model runs. Paid APIs, powerful GPUs, and distributed training are used only in optional extensions.
Наши преподаватели
Как проходит обучение
The course is self-paced and primarily text-based, with equations, illustrations, tables, code, and links to primary sources.
The core route contains 13 modules in three parts: foundations and a fast introduction to classical RL; LLM post-training and RL; infrastructure, evaluation, and safety.
Most modules follow a recurring learning route:
• a precise problem statement and data contract;
• the mechanism, equations, and small numerical checks;
• the mapping to LLMs and modern implementations;
• auto-graded assessments after the main sections;
• analytical and computational exercises;
• detailed solutions, a notebook, and key papers for further study.
Assessments are graded automatically. Regular exercises are not an external examination: attempt a problem first, then open the detailed solution.
Practice is divided into resource tiers. Tier A runs on CPU and Stepik; Tier B uses a free Colab T4; optional B+ and C work may require a paid API, a powerful GPU, or a distributed run.
The notebooks run in Colab or locally and separate inexpensive smoke checks from real branches using open models. Extended experiments require explicitly recording model and dataset versions, the config, seed, and metrics.
A study load of roughly five to seven hours per week is recommended, but there is no fixed schedule. Readers with classical RL experience can move through the first five modules more quickly.
Программа курса
Отзывы прошедших курс
Что вы получаете
- 13 modules and 114 lessons: 62 lecture lessons, 39 separate assignment lessons, and 13 primary-source lessons.
- 750 Stepik steps, including 414 assignments and self-checks; the RU and EN editions are synchronized, and lecture lessons contain clickable tables of contents.
- 151 self-checks embedded in the lectures, each with a prompt and an expandable worked solution.
- 109 conceptual tasks in native Stepik test formats, each worth 1 point and accompanied by an expandable explanation.
- 77 analytical and computational tasks with detailed solutions and explicit accepted tolerances wherever a numeric answer is required.
- 77 programming tasks for algorithm design and implementation, with expandable reference solutions and no mandatory code execution on the platform.
- 13 localized notebooks—one per module—attached to the relevant lectures and suitable for local Jupyter or Colab.
- Detailed derivations, negative cases, assumption checks, and failure-mode diagnostics instead of one-line answer comments.
- 13 separate Sources lessons with key papers, primary references, claim status, and limits on how far results can be transferred.
- A practical route from MDPs, bandits, REINFORCE, and PPO to reward models, RLHF/DPO, RLVR/GRPO, test-time search, agentic RL, infrastructure, evaluation, and safety.