The AI Memory Wall: Why LLMs & AI Accelerators Hit a Bottleneck
From IBM Technology · · 12 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.
If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.
Description
Learn more about the Memory Wall here → https://ibm.biz/~NjOWdJHJy
More GPU compute doesn't always mean faster AI models. Kaoutar El Maghraoui explains the memory wall, arithmetic intensity, and why LLM inference can become memory-bound or compute-bound.Learn how memory bandwidth, batching, and the roofline model impact AI performance.
00:00 — Why Faster Chips Don’t Always Make AI Workloads Faster 00:32 — The Memory Wall: The Hidden Bottleneck in Modern Computing 01:00 — Compute Growth vs Memory Bandwidth Explained 02:39 — Arithmetic Intensity: The Metric That Predicts Performance 03:48 — Understanding the Roofline Model for AI Optimization 05:27 — Compute-Bound vs Memory-Bound Workloads Explained 08:25 — LLM Decode vs Prefill: Why Performance Varies 09:48 — How Batching Improves AI Inference Efficiency 10:35 — Why Leading LLM Serving Systems Rely on Continuous Batching 11:22 — Practical Steps to Diagnose and Optimize AI Performance
Continue your education in the course The Accelerator Landscape on Coursera here → ibm.biz/~ELO03nsAl
AI news moves fast. Sign up for a monthly newsletter for AI updates from IBM → https://ibm.biz/~nnWVjebwV
AI was used in the creation of the transcript and metadata for this video.
#aimodels #aiaccelerator #ai #machinelearning --------------------------------------------------------------------------------------------------------- Find us on YouTube: 🔵 IBM Technology: https://www.youtube.com/@IBMTechnology 🔵 IBM: https://www.youtube.com/@IBM 🔵 IBM Developer: https://youtube.com/@IBMDeveloperAdvocates