Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alice's adventures in a differentiable wonderland

sscardapane.it

Post reply on HN
Print view View original

71–80 of 103 posts

Re: Alice's adventures in a differentiable wonderland

#71

Post by a_random_canuck »

p1esk wrote: post #63
xanderlewis wrote: post #61

Earlier quoted context omitted.

I don’t think the author literally meant tweaking the parameters and seeing what happens; it’s probably an analogy meant to give a sense of how the gradient indicates what direction and to what degree the parameters should be tweaked. Basically, substitute ‘the gradient is positive’ for ‘increasing this parameter decreases performance’ and vice versa and it becomes correct.

That substitution is the main difference between SGD and RWP. It’s like describing bubble sort when you meant to describe quick sort. Would not fly on an ML 101 exam, or in an ML job interview.

I don’t think anyone is trying to pass an exam here, but just give an understandable overview to a general audience.

Re: Alice's adventures in a differentiable wonderland

#72

Post by kragen »

makerdiety wrote: post #43
xanderlewis wrote: post #25

Earlier quoted context omitted.

It’s not ‘inaccurate’. The mark of true mastery is an ability to make terse statements that convey a huge amount without involving excessive formality or discussion of by-the-by technical details. If ever you’ve spoken to world-renowned experts in pure mathematics or other highly technical and pendantic fields, you’ll find they’ll say all sorts of ‘inaccurate’ things in conversation (or even in written documents). It…

Invoking excessive formality and discussions of minute technical details leads to a cathedral of knowledge built on autistic pedantry. The chosen rabbit hole to get lost in needs to be the correct one. And human science is riddled with the paths that have naive or childish fundamentals.

> a cathedral of knowledge built on autistic pedantry

this is certainly true, but more often we use its short name, 'math'. it turns out to be far more effective than so-called common sense

Re: Alice's adventures in a differentiable wonderland

#73

Post by jonas21 »

xanderlewis wrote: post #4

> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…

I feel like this statement is both obvious after spending a few minutes working with neural networks and completely useless in helping you build better neural networks.

It's kind of like saying, "Stripped of anything else, works of literature are compositions of words"

Re: Alice's adventures in a differentiable wonderland

#74

Post by kragen »

xanderlewis wrote: post #37
kragen wrote: post #31

Earlier quoted context omitted.

my experience with world-renowned experts in pure mathematics is that they are much more careful than the average bear to explicitly qualify inaccurate things as inaccurate, because their discipline requires them to be very clear about precisely what they are saying discontinuity of a function at x does not, according to the usual definition of 'continuity', require the function to have a value at x ; indeed, functio…

> they are much more careful than the average bear to explicitly qualify inaccurate things as inaccurate Sure. But what part of this entirely worded in natural language, and very short statement made you think it was a technical, formal statement? I think you’re just taking an opportunity to flex your knowledge of basic calculus, and deliberately attributing intent to the author that isn’t there in order to look clev…

you have an interesting point of view, and some of the things you have said are correct, but if you try to use gradient descent on a function from, say, ℤ → ℝ, you are going to be a very sad xanda. i would indeed describe such a function as being discontinuous not just at π but everywhere, at least with the usual definition of continuity (though there is a sense in which such a function could be, for example, scott-continuous)

even in the case of a single discontinuity in the derivative, like in relu', you lose the intermediate value theorem and everything that follows from it; it's not an inconsequential or marginally relevant fact

Re: Alice's adventures in a differentiable wonderland

#75

Post by kadushka »

captainclam wrote: post #69
kadushka wrote: post #59

Earlier quoted context omitted.

it comes down to "does increasing this parameter improve the performance on this task? Yes? Move the dial up a bit. No? Down a bit..." x 1e9 This is not how gradient based NN optimization works. What you described is called "random weight perturbation", a variant of evolutionary algorithms. It does not scale to networks larger than a few thousand parameters for obvious reasons. NNs are optimized by directly computing…

I guess you could say I don't know RWP from Adam! :D My og comment wasn't to accurately explain gradient optimization, I was just expressing a sentiment not especially aimed at experts and not especially requiring details. Though I'm afraid I subjected you to the same "cringe" I experience when I read pop sci/tech articles describe deep learning optimization as "the algorithm" being "rewarded" or "punished," haha.

No worries, we're all friends here!

it's just you happened to accidentally describe the idea behind RWP, which is a gradient-free optimization method, so I thought I should point it out.

Re: Alice's adventures in a differentiable wonderland

#76

Post by m463 »

xanderlewis wrote: post #4

> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…

chess is pattern matching.

Re: Alice's adventures in a differentiable wonderland

#77

Post by jj3 »

kragen wrote: post #74
xanderlewis wrote: post #37

Earlier quoted context omitted.

> they are much more careful than the average bear to explicitly qualify inaccurate things as inaccurate Sure. But what part of this entirely worded in natural language, and very short statement made you think it was a technical, formal statement? I think you’re just taking an opportunity to flex your knowledge of basic calculus, and deliberately attributing intent to the author that isn’t there in order to look clev…

you have an interesting point of view, and some of the things you have said are correct, but if you try to use gradient descent on a function from, say, ℤ → ℝ, you are going to be a very sad xanda. i would indeed describe such a function as being discontinuous not just at π but everywhere, at least with the usual definition of continuity (though there is a sense in which such a function could be, for example, scott-c…

Note that any function ℤ → ℝ is continuous on its domain but nowhere differentiable.

A Scott-continuous function ℤ → ℝ must be monontonous. So not every such function is Scott-continuous.

Re: Alice's adventures in a differentiable wonderland

#78

Post by srean »

phkahler wrote: post #30
xanderlewis wrote: post #4

> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…

>> one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural’ name And yet, artificial neural networks ARE an approximation of how biological neurons work. It is worth noting that they came out of neurobiology and not some math department - well at least in the forward direction, I'm not sure who came up with the training algorithms (probably the math folks). Should they be co…

> And yet, artificial neural networks ARE an approximation of how biological neurons work

For a non-vapid/non-vacuous definition of 'approximation' this is not true at all. It is well understood that (i) back-propagation is biologically infeasible in the brain (ii) output 'voltage' is a transformed weighted average of the input 'voltage' -- is not how neurons operate. (ii) is in the 'not even wrong' category.

Neurons operate in terms of spikes and frequency and quiescence of spiking. If you are interested any undergrad text in neurobiology will help correct the wrong notions.

Re: Alice's adventures in a differentiable wonderland

#79

Post by srean »

naasking wrote: post #44
xanderlewis wrote: post #40

Earlier quoted context omitted.

As the commenter below mentions, the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version. The neural network version is essentially just a weighted sum, with an extra layer of shaping applied afterwards to make it nonlinear. As far as I know, we still don’t understand all of the complexity about how biological neurons work. Even skimming the Wikipedia page for ‘neur…

> the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version This is a difference of degree not of kind, because neural networks are Turning complete. Whatever additional complexity the neuron has can itself be modelled as a neural network. Edit: meaning, that if the greater complexity of a biological neuron is relevant to its information processing component, then tha…

> This is a difference of degree not of kind

Nope.

Neurons in our brain operate fundamentally differently. They work by transient spikes and information is carried not by the intensity of the spike voltage, but by the frequency of spiking. This is a fundamentally different phenomenon than ANNs where the output (voltage) is a squash transformed aggregated input values (voltages).

Re: Alice's adventures in a differentiable wonderland

#80

Post by chriswarbo »

phkahler wrote: post #30
xanderlewis wrote: post #4

> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…

>> one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural’ name And yet, artificial neural networks ARE an approximation of how biological neurons work. It is worth noting that they came out of neurobiology and not some math department - well at least in the forward direction, I'm not sure who came up with the training algorithms (probably the math folks). Should they be co…

> And yet, artificial neural networks ARE an approximation of how biological neurons work.

Only if you limit yourself to "sums of weighted inputs, sent through a 1D activation function".

However, the parent said "differentiable primitives": these days people have built networks that contain differentiable ray-tracers, differentiable physics simulations, etc. Those seem like crazy ideas if we limit ourselves to the "neural" analogy; but are quite natural for a "composition of differentiable primitives" approach.

Post reply on HN