Skip to main content
AmigoFish

What Is Quantization? How AI Models Get Smaller & Faster

From IBM Technology · · 8 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.

If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.

Watch on YouTube

Not rated yet

Description

Learn more about Quantization here → https://ibm.biz/~NdqpU881V

Huge AI models come with huge memory requirements.Grace Ableidinger explains how quantization compresses LLMs using lower-precision formats like INT8 and INT4. Learn how model size, GPU memory, speed, and accuracy trade off during AI inference.

00:00 – Understand What Quantization Does 00:23 – See Why LLMs Need Quantization 00:51 – Connect AI Parameters to Memory Requirements 01:31 – Calculate the Memory Cost of a 70B LLM 02:13 – See How Quantization Reduces Precision 02:53 – Follow Quantization Layer by Layer 04:21 – Compare FP16, FP8, INT8 & INT4 05:34 – Balance Memory Savings with Model Accuracy 06:33 – Compare PTQ vs Quantization-Aware Training 07:17 – Choose the Right Quantization Tradeoff

AI news moves fast. Sign up for a monthly newsletter for AI updates from IBM → https://ibm.biz/~SLKUNTFaQ

AI was used in the creation of the transcript and metadata for this video.

#llm #aimodels #aiinference #machinelearning #aiengineering

--------------------------------------------------------------------------------------------------------- Find us on YouTube: 🔵 IBM Technology: https://www.youtube.com/@IBMTechnology 🔵 IBM: https://www.youtube.com/@IBM 🔵 IBM Developer: https://youtube.com/@IBMDeveloperAdvocates

Back to IBM Technology · Report a problem