Skip to main content
AmigoFish

[vLLM Office Hours #59] - Advanced techniques for model compression - October 22, 2026

From Red Hat · · 52 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.

If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.

Watch on YouTube

Not rated yet

Description

Welcome to vLLM office hours! These bi-weekly sessions are your chance to stay current with the vLLM ecosystem, ask questions, and hear directly from contributors and power users.

This week’s special topic: Advanced Techniques for Model Compression.

vLLM project update from core maintainer Michael Goin, covering a new reviewer role in the project’s governance, vLLM Metal for Apple silicon, watermarking, adaptive verification without confidence heads, early work on NVIDIA’s Vera Rubin, mega kernels for Kimi K3, and how vLLM stacks up against DeepSeek’s DeepInfer. Plus two releases, v0.31 and v0.32, which brought faster restarts and replica scale-out, a structured decisions API, wider NVFP4 KV cache support, and Model Runner V2 as the default on AMD GPUs.

Then our special topic with Kyle Sayers and Sawyer Bowerman (Red Hat AI) on getting more out of LLM Compressor. Kyle covers why compressing a frontier model is often harder on memory than serving one, and the sequential pipeline and data parallel schemes that make it tractable, along with the settings most users leave on the table. Sawyer walks through what observers actually do and how they affect quantization error, which quantization recipes are worth the extra calibration, and two approaches to expert pruning: REAP, which removes the least-used experts outright, and MoNE, which replaces them with cheaper stand-ins.

Slides: https://docs.google.com/presentation/d/1pLK_fOzAxUbtLQRnx-wZACFOXy16JEEHPk5Rjh0IeMs

Want to join the discussion live on Google Meet? Get a calendar invite at https://vllm.ai/office-hours

Timestamps: 00:00 Intro 02:21 vLLM community update 17:00 vLLM v0.31 and v0.32 releases 19:32 Advanced techniques for model compression 28:38 Observers, quantization recipes, and expert pruning 44:21 Q&A

Back to Red Hat · Report a problem