Skip to main content
AmigoFish

Training Your Own Agent Models With RL Is Now Practical - Jason Mancuso, Modal

From AI Engineer · · 20 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.

If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.

Watch on YouTube

Not rated yet

Description

Jason Mancuso of Modal argues that treating models as swappable backends is a losing strategy, and that teams with agents in production already have what they need for reinforcement learning: a harness, evals and real traces. He explains why RL generalizes better than fine-tuning on recorded traces, then walks through a stable recipe for asynchronous RL built on GRPO with masked importance sampling, score centering and top-p sampling. He shows how sparse weight updates let Modal spread rollouts across clouds, GPU types and quantization levels while training stays stable and rollouts get 15% cheaper.

Chapters: 0:00 Intro 0:20 Models aren't static backends 1:09 Why teams fine-tune 3:35 Why RL instead of SFT 4:46 You already have what you need 6:51 How reinforcement learning works 7:41 Async RL and staleness 9:38 A stable RL recipe 12:38 Why RL on Modal is different 14:55 Sparse weight updates 16:21 Mixing GPU types for rollouts 17:56 15% cheaper rollouts 19:16 Own your stack

Jason Mancuso https://modal.com

Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!

Back to AI Engineer · Report a problem

Training Your Own Agent Models With RL Is Now Practical - Jason Mancuso, Modal | AI Engineer | AmigoFish