Why Your Local AI Is Slow
From Tech With Tim · · 51 sec

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.
If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.
Description
Here's the thing nobody tells you about local AI speed: for every single token a model generates, it rereads itself entirely from memory. All 17GB, every token. That's why memory bandwidth sets a hard speed ceiling, and the formula is simple: bandwidth divided by model size.
On an M5 Max at 614 GB/s, a 17GB model predicts around 36 tokens per second, with real benchmarks closer to 31. It's not exact, but it gets you close to the max speed you'll hit before any optimization, useful to know before you buy.
#techwithtim #ai #machinelearning