Why You Can’t Run That AI Model
From Tech With Tim · · 45 sec

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.
If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.
Description
Why can't you run an AI model even when you have enough storage? Because an AI model is just billions of numbers, and "8B" means 8 billion parameters, not 8 gigabytes. At full precision (16 bits each), those 8 billion numbers take about 16GB.
Compress them to 4 bits each, called quantization, and you're down to roughly 4GB plus some overhead. The catch: the whole model has to sit in fast memory your GPU can reach, not your SSD, not your hard drive, not even regular system RAM. Having space to download a model doesn't mean you can actually run it.
#techwithtim #ai #machinelearning