Skip to main content
AmigoFish

Why You Can’t Run That AI Model

From Tech With Tim · · 45 sec

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.

If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.

Watch on YouTube

Not rated yet

Description

Why can't you run an AI model even when you have enough storage? Because an AI model is just billions of numbers, and "8B" means 8 billion parameters, not 8 gigabytes. At full precision (16 bits each), those 8 billion numbers take about 16GB.

Compress them to 4 bits each, called quantization, and you're down to roughly 4GB plus some overhead. The catch: the whole model has to sit in fast memory your GPU can reach, not your SSD, not your hard drive, not even regular system RAM. Having space to download a model doesn't mean you can actually run it.

#techwithtim #ai #machinelearning

Back to Tech With Tim · Report a problem

Why You Can’t Run That AI Model | Tech With Tim | AmigoFish