Skip to main content
AmigoFish

I Made My Local AI Pick Its Own Model

From Techno Tim · · 14 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.

If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.

Watch on YouTube

Not rated yet

Description

I wanted my local AI to choose the right model automatically instead of making me pick between speed and capability for every request.

I test NVIDIA Switchyard for local AI model routing with Nemotron 3.5 Lightning on an RTX 3090 and DeepSeek V4 Flash across two GX10s. I cover Stage routing, vLLM, context windows, and some of the problems I ran into while testing it with real coding agents.

In my best autonomous coding run, almost 88% of the routing decisions went to Nemotron while DeepSeek was still available when Switchyard decided it was needed.

This is pretty close to what I wanted. Most of the work stays on the fast local model, the more capable model gets pulled in when it makes sense, and I don't have to choose between them before I start.

Video Notes: https://technotim.com/posts/switchyard-local-ai-routing/

Merch Shop 🛍️: https://l.technotim.com/shop Support me on Patreon: https://www.patreon.com/technotim Sponsor me on GitHub: https://github.com/sponsors/timothystewart6 Subscribe on Twitch: https://www.twitch.tv/technotim Become a YouTube member: https://www.youtube.com/channel/UCOk-gHyjcWZNj3Br4oxwh0A/join Gear Recommendations: https://l.technotim.com/gear Get Help in Our Discord Community: https://l.technotim.com/discord 2nd channel: https://www.youtube.com/@TechnoTimTinkers

(Affiliate links may be included in this description. I may receive a small commission at no cost to you.)

00:00 Automatic AI Model Routing 01:09 Nemotron Lightning vs DeepSeek 01:40 Fixing Nemotron's Performance 02:35 Is Nemotron Actually Good? 02:58 How NVIDIA Switchyard Works 04:55 Putting Stage Routing to the Test 05:28 The 32K Context Problem 06:46 Fixing Routing with 128K Context 07:44 Testing on Real-World Code 08:40 Building the Autonomous Agent Test 09:46 Run 4: Getting Close 10:39 Run 5: 88% Routed to Nemotron 11:50 Run 6: When the Agent Got Stuck 12:11 Is NVIDIA Switchyard Worth It? 13:40 What's Next for My Local AI

Thank you for watching!

Back to Techno Tim · Report a problem