Skip to main content
AmigoFish

I built GPT-2 for $31.99

From Looking Glass Universe · · 10 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.

If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.

Watch on YouTube

Not rated yet

Description

The difference between this video and the last GPT I made is that that one was a toy model. It was trained on a very small set of data (the works of Shakespeare) and it doesn’t get very good. This one is much bigger, and reaches the level of GPT-2. It required quite a lot more optimisation to train it.

This is Karpathy's tutorial: https://youtu.be/l8pRSuU81PU?si=IZKoAl-YpqEn9Tyj

The code: https://github.com/karpathy/build-nanogpt/blob/master/train_gpt2.py

My write up as I went: https://colab.research.google.com/drive/1awhFM8oIGMVTQII-S7rsQGFeJDDxFoEV?usp=sharing

Back to Looking Glass Universe · Report a problem