Skip to main content
AmigoFish

Why Language Models Surprise You

From The Coding Train Β· Β· 35 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.

If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.

Watch on YouTube

Not rated yet

Description

In this video, I explain the softmax function and the concept of temperature in the context of language models and neural networks. I start by revisiting the Markov chain coding challenge, then break down how softmax converts raw neural network outputs (logits) into probabilities, and how temperature can be used to adjust those probabilities making outputs more random (higher temperature) or more deterministic (lower temperature). I also briefly cover top-K and top-P sampling. Everything is demonstrated with p5.js sketches and a transformers.js language model running in the browser. Code: https://thecodingtrain.com/tracks/transformers-js/transformers-js/softmax-temperature

πŸš€ Watch this video ad-free on Nebula https://nebula.tv/videos/codingtrain-softmax-and-temperature-explained

p5.js Web Editor Sketches: πŸ•ΉοΈ Softmax and Temperature Visualization: https://editor.p5js.org/codingtrain/sketches/G39vTMSkz πŸ•ΉοΈ Markov Chain with Temperature: https://editor.p5js.org/codingtrain/sketches/SXqlv6-zI πŸ•ΉοΈ Markov Chain with Softmax and Temperature: https://editor.p5js.org/codingtrain/sketches/WNvGNuJHD πŸ•ΉοΈ LLM with Temperature: https://editor.p5js.org/codingtrain/sketches/36caiecXD

πŸŽ₯ Previous: https://youtu.be/KR61bXsPlLU πŸŽ₯ Next: https://youtu.be/s2jm2Z22ibA

References: πŸ““ Softmax function on Wikipedia: https://en.wikipedia.org/wiki/Softmax_function πŸ““ Markov Chains explained visually: https://setosa.io/ev/markov-chains/ πŸ““ Web Trigrams by Chris Harrison: https://www.chrisharrison.net/index.php/Visualizations/WebTrigrams πŸ““ Euler's number on Wikipedia: https://en.wikipedia.org/wiki/E_(mathematical_constant) πŸ““ p5.js exp() reference: https://p5js.org/reference/p5/exp/ πŸ’» Transformers.js Documentation: https://huggingface.co/docs/transformers.js πŸ€— SmolLM2 1.7B on Hugging Face: https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B πŸ“• Nature of Code: Genetic Algorithms: https://natureofcode.com/genetic-algorithms/

Videos: πŸš‚ https://youtu.be/KR61bXsPlLU πŸš‚ https://youtu.be/eGFJ8vugIWA πŸš‚ https://youtu.be/ETphJASzYes πŸš‚ https://youtu.be/kwcillcWOg0

Related Coding Challenges: πŸš‚ https://youtu.be/eGFJ8vugIWA πŸš‚ https://youtu.be/s2jm2Z22ibA

Timestamps: 0:00:00 Hello! 0:02:50 Markov chains as language models 0:07:48 From logits to probabilities 0:10:47 The softmax formula 0:14:18 Implementing softmax in p5.js 0:15:59 Weighted selection (sampling) 0:20:20 Temperature 0:23:02 Applying softmax and temperature to the Markov chain 0:28:01 Temperature with a Transformers.js language model 0:31:55 Top-K and Top-P sampling 0:33:38 Goodbye!

Editing by Mathieu Blanchette Animations by Jason Heglund Music from Epidemic Sound

πŸš‚ Website: https://thecodingtrain.com/ πŸ‘Ύ Share Your Creation! https://thecodingtrain.com/guides/passenger-showcase-guide 🚩 Suggest Topics: https://github.com/CodingTrain/Suggestion-Box πŸ’‘ GitHub: https://github.com/CodingTrain πŸ’¬ Discord: https://thecodingtrain.com/discord πŸ’– Membership: http://youtube.com/thecodingtrain/join πŸ›’ Store: https://standard.tv/codingtrain πŸ–‹οΈ Twitter: https://twitter.com/thecodingtrain πŸ“Έ Instagram: https://www.instagram.com/the.coding.train/

πŸŽ₯ https://www.youtube.com/playlist?list=PLRqwX-V7Uu6ZiZxtDDRCi6uhfTH4FilpH πŸŽ₯ https://www.youtube.com/playlist?list=PLRqwX-V7Uu6Zy51Q-x9tMWIv9cueOFTFA

πŸ”— p5.js: https://p5js.org πŸ”— p5.js Web Editor: https://editor.p5js.org/ πŸ”— Processing: https://processing.org

πŸ“„ Code of Conduct: https://github.com/CodingTrain/Code-of-Conduct

This description was auto-generated. If you see a problem, please open an issue: https://github.com/CodingTrain/thecodingtrain.com/issues/new

#softmax #temperature #languagemodels #machinelearning #transformersjs #markovchain #ngram #logits #probability #weightedselection #sampling #topk #topp #neuralnetworks #eulersnumber #javascript #p5js #transformersjs

Back to The Coding Train Β· Report a problem