Skip to main content
AmigoFish

Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 5 - LLM Systems

From Stanford Online · · 1 hr 47 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.

If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.

Watch on YouTube

Not rated yet

Description

For more information about Stanford’s graduate programs, visit: https://stanford.io/grad-ed-cme295-A2026

To follow along with the course schedule and syllabus, visit: https://cme295.stanford.edu/syllabus/

Chapters: 00:00:00 Introduction 00:09:12 Flash attention 00:50:48 Distributed training 00:54:25 Data parallelism (DP) 00:55:30 Zero Redundancy Optimizer (ZeRO) 01:00:29 Tensor parallelism (TP) 01:02:05 Pipeline parallelism (PP) 01:04:12 Context parallelism (CP) 01:06:10 Expert parallelism (EP) 01:09:52 Inference optimizations 01:22:49 KV caching 01:24:30 Paged attention 01:28:01 Grouped-query attention (GQA) 01:30:43 Multi-head latent attention (MLA) 01:32:40 Speculative decoding 01:45:27 Multi-token prediction (MTP)

Afshine Amidi is an Adjunct Professor at Stanford University.

Shervine Amidi is an Adjunct Professor at Stanford University.

Follow the playlist: https://youtube.com/playlist?list=PLakGfbKCei80&si=HF2tNywj7nXIQq8X

Back to Stanford Online · Report a problem