Skip to main content
AmigoFish

Mastering AI Inference: Production vLLM Tunning

From Red Hat ·

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.

If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.

Watch on YouTube

Not rated yet

Description

Mastering Inference is a live, episode-by-episode tuning playbook for the vLLM and llm-d inference stack. One layer at a time, from quantization, sparsity, and speculative decoding to parallelism, GPU sizing, and routing. Every session bridges the gap between upstream capability and production reality, with practical takeaways you can apply to your own deployment.

In "Production vLLM Tuning," we start where most of the performance is: not in a redesign, but in the configuration you already have. We work through the model layer first (quantization, sparsity, and speculative decoding) then the engine: getting parallelism right, diagnosing where latency and throughput are actually lost under real production load, and right-sizing your GPUs to the workload you actually have, not the one you fear. You'll leave with a practical order of operations for your own deployment and a clear way to know when you've genuinely exhausted what configuration alone can give you, and a solid foundation to move towards desegregating serving.

Back to Red Hat · Report a problem