Skip to main content
AmigoFish

How AI Models Scale Beyond a Single GPU Across LLM Workloads

From IBM Technology · · 9 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.

If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.

Watch on YouTube

Not rated yet

Description

Learn more about AI Models here → https://ibm.biz/~GnXROtDog

The biggest AI models can't fit on one GPU. Grace Ableidinger explains how distributed AI inference scales LLMs across GPUs using data, pipeline, tensor, and expert parallelism. Learn how KV cache, throughput, and prefill/decode bottlenecks shape production AI serving.

00:00 – See Why AI Models Need Distributed Inference 01:50 – Scale LLM Traffic with Data Parallelism 02:45 – Split AI Models with Pipeline Parallelism 03:45 – Divide LLM Layers with Tensor Parallelism 04:32 – Scale Mixture-of-Experts Across GPUs 05:44 – Separate LLM Prefill and Decode Workloads 07:51 – Combine GPU Parallelism for Production AI 08:29 – Orchestrate Distributed LLM Inference

AI news moves fast. Sign up for a monthly newsletter for AI updates from IBM → https://ibm.biz/~86jhlfVag

AI was used in the creation of the transcript and metadata for this video.

#llm #aimodels #gpu #machinelearning

--------------------------------------------------------------------------------------------------------- Find us on YouTube: 🔵 IBM Technology: https://www.youtube.com/@IBMTechnology 🔵 IBM: https://www.youtube.com/@IBM 🔵 IBM Developer: https://youtube.com/@IBMDeveloperAdvocates

Back to IBM Technology · Report a problem