Alice's adventures in a differentiable wonderland
sscardapane.it
Alice's adventures in a differentiable wonderland
1–10 of 103 posts
Re: Alice's adventures in a differentiable wonderland
#2[dead]
Re: Alice's adventures in a differentiable wonderland
#3I wonder if the usage of Alice & Wonderland takes inspiration from Douglas Hofstadter's "Gödel, Escher, Bach: an Eternal Golden Braid" ?
Re: Alice's adventures in a differentiable wonderland
#4> Stripped of anything else, neural networks are compositions of differentiable primitives
I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence.
I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural’ name; deep learning is simply the application of sequences of operations that are nonlinear (and hence capable of encoding arbitrary complexity) but nonetheless differentiable and so efficiently optimisable.
Re: Alice's adventures in a differentiable wonderland
#5I wonder if the usage of Alice & Wonderland takes inspiration from Douglas Hofstadter's "Gödel, Escher, Bach: an Eternal Golden Braid" ?
It's a pretty common trope, especially for math-related books, e.g. Alex Bellos' "Alex's Adventures in Numberland".
Re: Alice's adventures in a differentiable wonderland
#6> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…
What does "differentiable primitives" mean here?
Re: Alice's adventures in a differentiable wonderland
#7> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…
What does "differentiable primitives" mean here?
Continuous mathematical functions which have derivatives.
Re: Alice's adventures in a differentiable wonderland
#8I wonder if the usage of Alice & Wonderland takes inspiration from Douglas Hofstadter's "Gödel, Escher, Bach: an Eternal Golden Braid" ?
No, I think the inspiration is more direct https://duckduckgo.com/?q=lewis+carroll+alice+in+wonderland+...
Re: Alice's adventures in a differentiable wonderland
#9> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…
What does "differentiable primitives" mean here?
I think it’s referring to ‘primitive functions’ in the sense that they’re the building blocks of more complicated functions. If f and g are differentiable, f+g, fg, f/g (as long as g is never zero)… and so on are differentiable too. Importantly, f composed with g is also differentiable, and so since the output of the whole network as a function of its input is a composition of these ‘primitives’ it’s differentiable too.
The actual primitive functions in this case would be things like the weighted sums of activations in the previous layer to get the activation of a given layer, and the actual ‘activation functions’ (traditionally something like a sigmoid function; these days a ReLU) associated with each layer.
‘Primitives’ is also sometimes used as a synonym for antiderivatives, but I don’t think that’s what it means here.
Edit: it just occurred to me from a comment below that you might have meant to ask what the ‘differentiable’ part means. See https://en.wikipedia.org/wiki/Differentiable_function.