better remove all the Disney-based Alice in Wonderland character intellectual property from the book.
Alice's adventures in a differentiable wonderland
11–20 of 103 posts
Re: Alice's adventures in a differentiable wonderland
#12> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…
Ugh, exactly, it's so cool. I've been a deep learning practitioner for ~3 years now, and I feel like this notion has really been impressed upon me only recently.
I've spent an awful lot of mental energy trying to conceive of how these things work, when really it comes down to "does increasing this parameter improve the performance on this task? Yes? Move the dial up a bit. No? Down a bit..." x 1e9.
And the cool part is that this yields such rich, interesting, sometimes even useful, structures!
I like to think of this cognitive primitive as the analogue to the idea that thermodynamics is just the sum of particles bumping into each other. At the end of the day, that really is just it, but the collective behavior is something else entirely.
Re: Alice's adventures in a differentiable wonderland
#13> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…
Ugh, exactly, it's so cool. I've been a deep learning practitioner for ~3 years now, and I feel like this notion has really been impressed upon me only recently. I've spent an awful lot of mental energy trying to conceive of how these things work, when really it comes down to "does increasing this parameter improve the performance on this task? Yes? Move the dial up a bit. No? Down a bit..." x 1e9. And the cool part…
> At the end of the day, that really is just it, but the collective behavior is something else entirely.
Exactly. It’s not to say that neat descriptions like this are the end of the story (or even the beginning of it). If they were, there would be no need for this entire field of study.
But they are cool, and can give you a really clear conceptualisation of something that can appear more like a sum of disjoint observations and ad hoc tricks than a discipline based on a few deep principles.
Re: Alice's adventures in a differentiable wonderland
#14I wonder if the usage of Alice & Wonderland takes inspiration from Douglas Hofstadter's "Gödel, Escher, Bach: an Eternal Golden Braid" ?
Lewis Carrol's Alice in Wonderland features a number of logical and mathematical puzzles [1]
He also wrote What the Tortoise Said to Achilles (1895) in which the paradoxes of Zeno are discussed.
So it's more correct to say that GEB and this article are originally inspired by Lewis Carrol's work.
[1] I wrote a short article for my university magazine a long time ago. Some interesting references at the end https://abd.tiddlyspot.com/#%5B%5BMathematical%20Adventures%...
Re: Alice's adventures in a differentiable wonderland
#15Re: Alice's adventures in a differentiable wonderland
#16> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…
Ugh, exactly, it's so cool. I've been a deep learning practitioner for ~3 years now, and I feel like this notion has really been impressed upon me only recently. I've spent an awful lot of mental energy trying to conceive of how these things work, when really it comes down to "does increasing this parameter improve the performance on this task? Yes? Move the dial up a bit. No? Down a bit..." x 1e9. And the cool part…
NAND gates by themselves are kind of dull, but it's pretty cool what you can do with a billion of them.
Re: Alice's adventures in a differentiable wonderland
#17And then you learn about binary or ternary networks where gradients don’t really exist anywhere, and you start to wonder about the importance of this differentiability.
Re: Alice's adventures in a differentiable wonderland
#18In the literature, they're usually called convolutional layers (I think you can pretty much search and replace all uses of "convolutive" in the text).
Re: Alice's adventures in a differentiable wonderland
#19> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…
Before the recent AI boom, I was mystified by the possibility of AI and emulating humans (in no small part thanks to works of fiction showing AI powered androids). Then I created and trained some neural networks. Smaller ones, doing much of nothing special. That was enough to break the mysticism. To realize it was just multiplying matrices. Training them was a bit more advanced, but still applied mathematics.
Only recently have I begun to appreciate that the simplicity of the operation, applied to a large enough matrices, may still capture enough of the nature of intelligence and sentience. In the end we can be broken down into (relatively) simple chemical reactions, and it is the massive scale of these reactions that create real intelligence and sentience.
Re: Alice's adventures in a differentiable wonderland
#20> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…
It is soothing to the mind because it conveys that it’s understandable but it doesn’t take away from the complexity. You still have to read through math and pytorch code and debug nonsensical CUDA errors, comb through the data, etc etc