Skip to main content
AmigoFish

Building an LLM fine-tuning Dataset

From sentdex · · 1 hr 1 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.

If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.

Watch on YouTube

Not rated yet

Description

Going through the building of a QLoRA fine-tuning dataset for a language model. NVIDIA GTC signup: https://nvda.ws/3XTqlB6

Fine-tuning code: https://github.com/Sentdex/LLM-Finetuning 5000-step Walls1337bot adapter: https://huggingface.co/Sentdex/Walls1337bot-Llama2-7B-003.005.5000 WSB Dataset: https://huggingface.co/datasets/Sentdex/WSB-003.005 "I have every reddit comment" original reddit post and torrent info: https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/ 2007-2015 Reddit Archive.org: https://archive.org/download/2015_reddit_comments_corpus/reddit_data/ Reddit BigQuery 2007-2019 (this has other data besides reddit comments too!): https://reddit.com/r/bigquery/comments/3cej2b/17_billion_reddit_comments_loaded_on_bigquery/

Contents:

0:00 - Introduction to Dataset building for fine-tuning. 02:53 - The Reddit dataset options (Torrent, Archive.org, BigQuery) 06:07 - Exporting BigQuery Reddit (and some other data) 14:44 - Decompressing all of the gzip archives 25:13 - Re-combining the archives for target subreddits 28:29 - How to structure the data 40:40 - Building training samples and saving to database 48:49 - Creating customized training json files 54:11 - QLoRA training and results

Neural Networks from Scratch book: https://nnfs.io Channel membership: https://www.youtube.com/channel/UCfzlCWGWYyIQ0aLC5w48gBQ/join Discord: https://discord.gg/sentdex Reddit: https://www.reddit.com/r/sentdex/ Support the content: https://pythonprogramming.net/support-donate/ Twitter: https://twitter.com/sentdex Instagram: https://instagram.com/sentdex Facebook: https://www.facebook.com/pythonprogramming.net/ Twitch: https://www.twitch.tv/sentdex

Back to sentdex · Report a problem