Skip to main content
AmigoFish

Reinventing Entropy | Compression is Intelligence Part 1

From 3Blue1Brown · · 32 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.

If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.

Watch on YouTube

Not rated yet

Description

How to study the compressibility of language. Check out our virtual career fair: https://3b1b.co/talent See new projects before they go live: https://3b1b.co/support

Animation credit: Manim scenes by Aaron Gostein and Grant Sanderson Shannon’s story, as well as those for various pi creatures, by Mitchell Zemil. Lunar robot and prediction/compression coin by Paul Dancstep NanoGPT animations by Clayton Rabideau

The way of visualizing entropy shown here is something I first came across in this excellent post by Chris Olah: https://colah.github.io/posts/2015-09-Visual-Information/

Shannon’s “A Mathematical Theory of Communication” https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf

Shannon’s “Prediction and Entropy of Printed English” https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf

Scientific American article that mentions the story with Von Neumann suggesting the name Entropy: https://www.esalq.usp.br/lepse/imgs/conteudo_thumb/Energy-and-Information.pdf

Timestamps:

0:00 - On “Compression is intelligence.” 3:28 - The warmup example 10:46 - What perfect compression looks like 14:47 - Defining information 17:40 - Information of language 24:29 - Defining Entropy 31:14 - 3b1b Talent

------------------

These animations are largely made using a custom Python library, manim. See the FAQ comments here: https://3b1b.co/faq#manim

Music by Vincent Rubinetti. https://vincerubinetti.bandcamp.com/album/the-music-of-3blue1brown https://open.spotify.com/album/1dVyjwS8FBqXhRunaG5W5u

------------------

3blue1brown is a channel about animating math, in all senses of the word animate. If you're reading the bottom of a video description, I'm guessing you're more interested than the average viewer in lessons here. It would mean a lot to me if you chose to stay up to date on new ones, either by subscribing here on YouTube or otherwise following on whichever platform below you check most regularly.

Mailing list: https://3blue1brown.substack.com Twitter: https://twitter.com/3blue1brown Bluesky: https://bsky.app/profile/3blue1brown.com Instagram: https://www.instagram.com/3blue1brown Reddit: https://www.reddit.com/r/3blue1brown Facebook: https://www.facebook.com/3blue1brown Patreon: https://patreon.com/3blue1brown Website: https://www.3blue1brown.com

Back to 3Blue1Brown · Report a problem