Skip to main content
AmigoFish

The AI Memory Wall: Why LLMs & AI Accelerators Hit a Bottleneck

From IBM Technology · · 12 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.

If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.

Watch on YouTube

Not rated yet

Description

Learn more about the Memory Wall here → https://ibm.biz/~NjOWdJHJy

More GPU compute doesn't always mean faster AI models. Kaoutar El Maghraoui explains the memory wall, arithmetic intensity, and why LLM inference can become memory-bound or compute-bound.Learn how memory bandwidth, batching, and the roofline model impact AI performance.

00:00 — Why Faster Chips Don’t Always Make AI Workloads Faster 00:32 — The Memory Wall: The Hidden Bottleneck in Modern Computing 01:00 — Compute Growth vs Memory Bandwidth Explained 02:39 — Arithmetic Intensity: The Metric That Predicts Performance 03:48 — Understanding the Roofline Model for AI Optimization 05:27 — Compute-Bound vs Memory-Bound Workloads Explained 08:25 — LLM Decode vs Prefill: Why Performance Varies 09:48 — How Batching Improves AI Inference Efficiency 10:35 — Why Leading LLM Serving Systems Rely on Continuous Batching 11:22 — Practical Steps to Diagnose and Optimize AI Performance

Continue your education in the course The Accelerator Landscape on Coursera here → ibm.biz/~ELO03nsAl

AI news moves fast. Sign up for a monthly newsletter for AI updates from IBM → https://ibm.biz/~nnWVjebwV

AI was used in the creation of the transcript and metadata for this video.

#aimodels #aiaccelerator #ai #machinelearning --------------------------------------------------------------------------------------------------------- Find us on YouTube: 🔵 IBM Technology: https://www.youtube.com/@IBMTechnology 🔵 IBM: https://www.youtube.com/@IBM 🔵 IBM Developer: https://youtube.com/@IBMDeveloperAdvocates

Back to IBM Technology · Report a problem