Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 5 - LLM Systems
From Stanford Online · · 1 hr 47 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.
If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.
Description
For more information about Stanford’s graduate programs, visit: https://stanford.io/grad-ed-cme295-A2026
To follow along with the course schedule and syllabus, visit: https://cme295.stanford.edu/syllabus/
Chapters: 00:00:00 Introduction 00:09:12 Flash attention 00:50:48 Distributed training 00:54:25 Data parallelism (DP) 00:55:30 Zero Redundancy Optimizer (ZeRO) 01:00:29 Tensor parallelism (TP) 01:02:05 Pipeline parallelism (PP) 01:04:12 Context parallelism (CP) 01:06:10 Expert parallelism (EP) 01:09:52 Inference optimizations 01:22:49 KV caching 01:24:30 Paged attention 01:28:01 Grouped-query attention (GQA) 01:30:43 Multi-head latent attention (MLA) 01:32:40 Speculative decoding 01:45:27 Multi-token prediction (MTP)
Afshine Amidi is an Adjunct Professor at Stanford University.
Shervine Amidi is an Adjunct Professor at Stanford University.
Follow the playlist: https://youtube.com/playlist?list=PLakGfbKCei80&si=HF2tNywj7nXIQq8X