Skip to main content
AmigoFish

Faster LLM Inference on Jetson With Speculative Decoding

From NVIDIA Developer · · 36 sec

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.

If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.

Watch on YouTube

Not rated yet

Description

How does speculative decoding speed up LLM inference on NVIDIA Jetson? Watch Qwen3.5 9B NVFP4 generate responses with speculative decoding enabled and disabled.

By drafting multiple tokens for the target model to verify together, speculative decoding can reduce sequential generation steps and deliver faster responses. Results depend on the model, hardware, and workload.

➡️ Learn more: https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference 📝 Tech blog: https://developer.nvidia.com/blog/frontier-reasoning-reaches-the-edge-how-to-deploy-and-optimize-models-on-nvidia-jetson ➡️ Explore models and turotials: https://www.jetson-ai-lab.com/

Back to NVIDIA Developer · Report a problem