Alice's adventures in a differentiable wonderland • Hacker News Board

Re: Alice's adventures in a differentiable wonderland

#31

Post by kragen » Tue, Apr 30, 2024, 8:09 PM UTC

xanderlewis wrote: post #25

jxy wrote: post #23

Earlier quoted context omitted.

> > Stripped of anything else, neural networks are compositions of differentiable primitives > I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. And I hate inaccurate statements like this. It pretends to be rigorous mathematical, but really just propagates erroneous information, and makes the whole article so much m…

It’s not ‘inaccurate’. The mark of true mastery is an ability to make terse statements that convey a huge amount without involving excessive formality or discussion of by-the-by technical details. If ever you’ve spoken to world-renowned experts in pure mathematics or other highly technical and pendantic fields, you’ll find they’ll say all sorts of ‘inaccurate’ things in conversation (or even in written documents). It…

my experience with world-renowned experts in pure mathematics is that they are much more careful than the average bear to explicitly qualify inaccurate things as inaccurate, because their discipline requires them to be very clear about precisely what they are saying

discontinuity of a function at x does not, according to the usual definition of 'continuity', require the function to have a value at x; indeed, functions that fail to have a value at x are necessarily discontinuous there, precisely because (as you say) they are not continuous there. https://en.wikipedia.org/wiki/Continuous_function#Definition...

there are other definitions of 'discontinuous' in use, but i can't think of one that would give the result you claim

Re: Alice's adventures in a differentiable wonderland

#32

Post by kragen » Tue, Apr 30, 2024, 8:16 PM UTC

phkahler wrote: post #30

xanderlewis wrote: post #4

> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…

>> one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural’ name And yet, artificial neural networks ARE an approximation of how biological neurons work. It is worth noting that they came out of neurobiology and not some math department - well at least in the forward direction, I'm not sure who came up with the training algorithms (probably the math folks). Should they be co…

anns originated in hypotheses about how neurobiology might work in the 01940s but diverged completely from neurobiology in the 01960s; they contain nothing we've learned about neurons in the last 50 years, and not much from before that either (they don't, for example, do hebbian learning). current anns use training methods like gradient descent with momentum and activation functions like relu which have no plausible biological realization

artificial neural networks are an approximation of biological neural networks in the same way that a submarine is an approximation of a fish

Re: Alice's adventures in a differentiable wonderland

#33

Post by gessha » Tue, Apr 30, 2024, 8:36 PM UTC

whimsicalism wrote: post #27

gessha wrote: post #20

Earlier quoted context omitted.

It is soothing to the mind because it conveys that it’s understandable but it doesn’t take away from the complexity. You still have to read through math and pytorch code and debug nonsensical CUDA errors, comb through the data, etc etc

the complexity is in the values learned from the optimization. even the pytorch code for a simple transformer is not that complex, attention is a simple mechanism, etc.

Complexity also comes from the number of papers that work out how different elements of network work and how to intuitively change them.

Why do we use conv operators, why do we use attention operators, when do we use one over the other? What augmentations do you use, how big of a dataset do you need, how do you collect the dataset, etc etc etc

Re: Alice's adventures in a differentiable wonderland

#34

Post by whimsicalism » Tue, Apr 30, 2024, 8:41 PM UTC

gessha wrote: post #33

whimsicalism wrote: post #27

Earlier quoted context omitted.

the complexity is in the values learned from the optimization. even the pytorch code for a simple transformer is not that complex, attention is a simple mechanism, etc.

Complexity also comes from the number of papers that work out how different elements of network work and how to intuitively change them. Why do we use conv operators, why do we use attention operators, when do we use one over the other? What augmentations do you use, how big of a dataset do you need, how do you collect the dataset, etc etc etc

idk, just using attention and massive web crawls gets you pretty far. a lot of the rest is more product-style decisions about what personality you want your LM to take.

I fundamentally don't think this technology is that complex.

Re: Alice's adventures in a differentiable wonderland

#35

Post by ubj » Tue, Apr 30, 2024, 8:58 PM UTC

p1esk wrote: post #17

And then you learn about binary or ternary networks where gradients don’t really exist anywhere, and you start to wonder about the importance of this differentiability.

...And then you start learning about generalizations of the notion of "gradient" to scenarios where the classical gradient doesn't exist :)

Re: Alice's adventures in a differentiable wonderland

#36

Post by jimbokun » Tue, Apr 30, 2024, 9:06 PM UTC

xanderlewis wrote: post #4

> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…

I always get the impression even the proponents of these algorithms when they didn't seem so promising, are shocked at the capabilities demonstrated by models built with such a relatively simple procedure.

Re: Alice's adventures in a differentiable wonderland

#37

Post by xanderlewis » Tue, Apr 30, 2024, 9:07 PM UTC

kragen wrote: post #31

xanderlewis wrote: post #25

Earlier quoted context omitted.

It’s not ‘inaccurate’. The mark of true mastery is an ability to make terse statements that convey a huge amount without involving excessive formality or discussion of by-the-by technical details. If ever you’ve spoken to world-renowned experts in pure mathematics or other highly technical and pendantic fields, you’ll find they’ll say all sorts of ‘inaccurate’ things in conversation (or even in written documents). It…

my experience with world-renowned experts in pure mathematics is that they are much more careful than the average bear to explicitly qualify inaccurate things as inaccurate, because their discipline requires them to be very clear about precisely what they are saying discontinuity of a function at x does not, according to the usual definition of 'continuity', require the function to have a value at x ; indeed, functio…

> they are much more careful than the average bear to explicitly qualify inaccurate things as inaccurate

Sure. But what part of this entirely worded in natural language, and very short statement made you think it was a technical, formal statement? I think you’re just taking an opportunity to flex your knowledge of basic calculus, and deliberately attributing intent to the author that isn’t there in order to look clever.

Regarding a function being discontinuous at a point outside its domain: if you take a completely naive view of what ‘discontinuous’ means, then I suppose you can say so. But discontinuity is just the logical negation of continuity. Observe:

To say that f: X —> Y (in this context, a real-valued function of real numbers) is continuous means precisely

∀x∈X ∀ε>0 ∃δ>0 |x - p| and so its negation looks like

∃x∈X ⌐ …

that is, there is a point in X, the domain of f where continuity fails.

For example, you wouldn’t talk about a function defined on the integers being discontinuous at pi, would you? That would just be weird.

To prove the point further, observe that the set of discontinuities (according to your definition) of any given function would actually include every number… in fact every mathematical object in the universe — which would make it not even a set in ZFC. So it’s absurd.

Even more reasons to believe functions can only be discontinuous at points of their domain: a function is said to be discontinuous if it has at least one discontinuity. By your definition, every function is discontinuous.

…anyway, I said we were going to be petty. I’m trying to demonstrate this is a waste of time by wasting my own time.

Re: Alice's adventures in a differentiable wonderland

#38

Post by OJFord » Tue, Apr 30, 2024, 9:09 PM UTC

andoando wrote: post #28

xanderlewis wrote: post #9

Earlier quoted context omitted.

I think it’s referring to ‘primitive functions’ in the sense that they’re the building blocks of more complicated functions. If f and g are differentiable, f+g, fg, f/g (as long as g is never zero)… and so on are differentiable too. Importantly, f composed with g is also differentiable, and so since the output of the whole network as a function of its input is a composition of these ‘primitives’ it’s differentiable t…

Is this function composition essentially lambda calculus then?

Function composition is just f(g(x)), considered as a single function that's the composition of f and g; it has the domain of f and the range of g.

In lambda calculus terminology it's an 'application' (with a function argument).

Re: Alice's adventures in a differentiable wonderland

#39

Post by laingc » Tue, Apr 30, 2024, 9:16 PM UTC

xanderlewis wrote: post #37

kragen wrote: post #31

Earlier quoted context omitted.

my experience with world-renowned experts in pure mathematics is that they are much more careful than the average bear to explicitly qualify inaccurate things as inaccurate, because their discipline requires them to be very clear about precisely what they are saying discontinuity of a function at x does not, according to the usual definition of 'continuity', require the function to have a value at x ; indeed, functio…

> they are much more careful than the average bear to explicitly qualify inaccurate things as inaccurate Sure. But what part of this entirely worded in natural language, and very short statement made you think it was a technical, formal statement? I think you’re just taking an opportunity to flex your knowledge of basic calculus, and deliberately attributing intent to the author that isn’t there in order to look clev…

Because memes aren't allowed on HN, you're not allowed to reply with the "akssshuallllyyy" meme, so you had to go to these lengths.

¯\_(ツ)_/¯

Re: Alice's adventures in a differentiable wonderland

#40

Post by xanderlewis » Tue, Apr 30, 2024, 9:18 PM UTC

phkahler wrote: post #30

xanderlewis wrote: post #4

> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…

>> one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural’ name And yet, artificial neural networks ARE an approximation of how biological neurons work. It is worth noting that they came out of neurobiology and not some math department - well at least in the forward direction, I'm not sure who came up with the training algorithms (probably the math folks). Should they be co…

As the commenter below mentions, the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version. The neural network version is essentially just a weighted sum, with an extra layer of shaping applied afterwards to make it nonlinear. As far as I know, we still don’t understand all of the complexity about how biological neurons work. Even skimming the Wikipedia page for ‘neuron’ will give you some idea.

The original idea of approximating something like a neuron using a weighted sum (which is a fairly obvious idea, given the initial discovery that neurons become ‘activated’ and they do so in proportion to how much the neurons they are connected to are) did come from thinking about biological brains, but the mathematical building blocks are incredibly simple and are hundreds of years old, if not thousands.