Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alice's adventures in a differentiable wonderland

sscardapane.it

Post reply on HN
Print view View original

41–50 of 103 posts

Re: Alice's adventures in a differentiable wonderland

#41

Post by xanderlewis »

laingc wrote: post #39
xanderlewis wrote: post #37

Earlier quoted context omitted.

> they are much more careful than the average bear to explicitly qualify inaccurate things as inaccurate Sure. But what part of this entirely worded in natural language, and very short statement made you think it was a technical, formal statement? I think you’re just taking an opportunity to flex your knowledge of basic calculus, and deliberately attributing intent to the author that isn’t there in order to look clev…

Because memes aren't allowed on HN, you're not allowed to reply with the "akssshuallllyyy" meme, so you had to go to these lengths. ¯\_(ツ)_/¯

You’re actually not far off. I’m somewhat embarrassed by the above, but I think it makes the point.

Re: Alice's adventures in a differentiable wonderland

#42

Post by xanderlewis »

andoando wrote: post #28
xanderlewis wrote: post #9

Earlier quoted context omitted.

I think it’s referring to ‘primitive functions’ in the sense that they’re the building blocks of more complicated functions. If f and g are differentiable, f+g, fg, f/g (as long as g is never zero)… and so on are differentiable too. Importantly, f composed with g is also differentiable, and so since the output of the whole network as a function of its input is a composition of these ‘primitives’ it’s differentiable t…

Is this function composition essentially lambda calculus then?

Composition here just means what it does for any two functions: the value of the ‘composition’ of f and g at x is defined to be f applied to g applied to x. In symbols, its: f∘g := f(g(x)) for each x in the domain of f. It may seem obvious, but the fact that this new thing is also a function (that is, its value is well-defined for every input) is actually a very useful thing indeed and leads to… well, most of mathematics.

You can certainly do function composition in lambda calculus: in fact, the act of composition itself is a higher order function (takes functions and returns a function) and you can certainly express it formally with lambda terms and such. It’s not really got anything to do with any particular language or model of computation though.

Re: Alice's adventures in a differentiable wonderland

#43

Post by makerdiety »

xanderlewis wrote: post #25
jxy wrote: post #23

Earlier quoted context omitted.

> > Stripped of anything else, neural networks are compositions of differentiable primitives > I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. And I hate inaccurate statements like this. It pretends to be rigorous mathematical, but really just propagates erroneous information, and makes the whole article so much m…

It’s not ‘inaccurate’. The mark of true mastery is an ability to make terse statements that convey a huge amount without involving excessive formality or discussion of by-the-by technical details. If ever you’ve spoken to world-renowned experts in pure mathematics or other highly technical and pendantic fields, you’ll find they’ll say all sorts of ‘inaccurate’ things in conversation (or even in written documents). It…

Invoking excessive formality and discussions of minute technical details leads to a cathedral of knowledge built on autistic pedantry. The chosen rabbit hole to get lost in needs to be the correct one. And human science is riddled with the paths that have naive or childish fundamentals.

Re: Alice's adventures in a differentiable wonderland

#44

Post by naasking »

xanderlewis wrote: post #40
phkahler wrote: post #30

Earlier quoted context omitted.

>> one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural’ name And yet, artificial neural networks ARE an approximation of how biological neurons work. It is worth noting that they came out of neurobiology and not some math department - well at least in the forward direction, I'm not sure who came up with the training algorithms (probably the math folks). Should they be co…

As the commenter below mentions, the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version. The neural network version is essentially just a weighted sum, with an extra layer of shaping applied afterwards to make it nonlinear. As far as I know, we still don’t understand all of the complexity about how biological neurons work. Even skimming the Wikipedia page for ‘neur…

> the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version

This is a difference of degree not of kind, because neural networks are Turning complete. Whatever additional complexity the neuron has can itself be modelled as a neural network.

Edit: meaning, that if the greater complexity of a biological neuron is relevant to its information processing component, then that just increases the number of artificial neural network neurons needed to describe it, it does not need any computation of a different kind.

Re: Alice's adventures in a differentiable wonderland

#45

Post by naasking »

xanderlewis wrote: post #4

> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…

> one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural’ name

One also shouldn't fall into the dual trap of assuming that just because one understands how a model works, it cannot have any bearing on the ever-mysterious operation of the brain.

Re: Alice's adventures in a differentiable wonderland

#46

Post by sideshowb »

xanderlewis wrote: post #4

> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…

> deep learning is simply the application of sequences of operations that are nonlinear but nonetheless differentiable

Though other things fit this description which are not deep learning. Like (shameless plug) my recent paper here https://ieeexplore.ieee.org/document/10497907

Re: Alice's adventures in a differentiable wonderland

#47

Post by andoando »

xanderlewis wrote: post #42
andoando wrote: post #28

Earlier quoted context omitted.

Is this function composition essentially lambda calculus then?

Composition here just means what it does for any two functions: the value of the ‘composition’ of f and g at x is defined to be f applied to g applied to x . In symbols, its: f∘g := f(g(x)) for each x in the domain of f . It may seem obvious, but the fact that this new thing is also a function (that is, its value is well-defined for every input) is actually a very useful thing indeed and leads to… well, most of mathe…

I didn't form my question too well. I understand all that. What I am asking is, are these function compositions equivalent to equivalent/similar to functions in lambda calculus?

I guess my question, is what are the primitive functions here doing?

Re: Alice's adventures in a differentiable wonderland

#48

Post by xanderlewis »

naasking wrote: post #44
xanderlewis wrote: post #40

Earlier quoted context omitted.

As the commenter below mentions, the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version. The neural network version is essentially just a weighted sum, with an extra layer of shaping applied afterwards to make it nonlinear. As far as I know, we still don’t understand all of the complexity about how biological neurons work. Even skimming the Wikipedia page for ‘neur…

> the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version This is a difference of degree not of kind, because neural networks are Turning complete. Whatever additional complexity the neuron has can itself be modelled as a neural network. Edit: meaning, that if the greater complexity of a biological neuron is relevant to its information processing component, then tha…

PowerPoint is Turing complete. Does that mean PowerPoint should be regarded as being biological or at least neuroscience-inspired?

Re: Alice's adventures in a differentiable wonderland

#49

Post by mistermann »

makerdiety wrote: post #43
xanderlewis wrote: post #25

Earlier quoted context omitted.

It’s not ‘inaccurate’. The mark of true mastery is an ability to make terse statements that convey a huge amount without involving excessive formality or discussion of by-the-by technical details. If ever you’ve spoken to world-renowned experts in pure mathematics or other highly technical and pendantic fields, you’ll find they’ll say all sorts of ‘inaccurate’ things in conversation (or even in written documents). It…

Invoking excessive formality and discussions of minute technical details leads to a cathedral of knowledge built on autistic pedantry. The chosen rabbit hole to get lost in needs to be the correct one. And human science is riddled with the paths that have naive or childish fundamentals.

This comment makes me want to both upvote and downvote with extreme enthusiasm/fury!

The sign of a truly good conversation?

Re: Alice's adventures in a differentiable wonderland

#50

Post by naasking »

SkyBelow wrote: post #19
xanderlewis wrote: post #4

> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…

Before the recent AI boom, I was mystified by the possibility of AI and emulating humans (in no small part thanks to works of fiction showing AI powered androids). Then I created and trained some neural networks. Smaller ones, doing much of nothing special. That was enough to break the mysticism. To realize it was just multiplying matrices. Training them was a bit more advanced, but still applied mathematics. Only re…

Exactly, the people who are derisive of those who consider ML models to exhibit glimmers of true intelligence because it's only matrix multiplications always amuse me. It's like they don't even realize the contradiction in holding the position that seemingly complex and intelligent outward behaviour should not be used as an indication of actual complexity and intelligence.
Post reply on HN