The GELU, SiLU and SwiGLU activation functions, clearly explained!!!
From StatQuest with Josh Starmer · · 24 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.
If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.
Description
There's a new activation function in town. Actually, there are a bunch of new activation functions in town. This StatQuest gives you the essential information you need at the very start, followed by a surprisingly interesting derivation of each of these new activation functions. It turns out that Dropout is all you need. Who knew?
If you'd like to support StatQuest, please consider... Patreon: https://www.patreon.com/statquest ...or... YouTube Membership: https://www.youtube.com/channel/UCtYLUTtgS3k1Fg4y5tAhLbw/join
...buying a book, a study guide, a t-shirt or hoodie, or a song from the StatQuest store... https://statquest.org/statquest-store/
...or just donating to StatQuest! paypal: https://www.paypal.me/statquest venmo: @JoshStarmer
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter: https://twitter.com/joshuastarmer
0:00 Awesome song and introduction 0:45 TL;DW (too long, didn't watch) 3:06 Background and why the Sigmoid and ReLU activation functions fail 9:18 How dropout improves training 11:16 GELU = Input dependent dropout 16:54 General equation for a class of activation functions 18:13 SiLU = Input dependent dropout 19:20 SwiGLU = Input dependent dropout with two extra weights.
#StatQuest