Alice's adventures in a differentiable wonderland • Hacker News Board

Re: Alice's adventures in a differentiable wonderland

#101

Post by citizen_friend » Thu, May 02, 2024, 4:20 AM UTC

naasking wrote: post #100

citizen_friend wrote: post #99

Earlier quoted context omitted.

> Taylor expansions That’s not what I’m talking about. This is a basic analysis topic: https://en.m.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_th... At least mid 1800s for a proof. 1700s also explored Fourier series. > stand by my position And you’re still ignoring the cybernetics, and perceptrons movement I keep referring to which was more than 100 years ago, and informed by Turing.

> That’s not what I’m talking about. This is a basic analysis topic: It's the same basic flaw: requiring continuous functions. Not all functions are continuous, therefore this is not sufficient. > And you’re still ignoring the cybernetics, and perceptrons movement I keep referring to which was more than 100 years ago, and informed by Turing. What about them? As long as they're universal, they can all simulate brains.…

This implies it works piecewise. That’s also true of neural nets lol. You have to keep adding more neurons to get the granularity of whatever your discontinuities are.

It’s also a different reason than Taylor series which uses differentiability.

You do not understand this subject. Please read before repeating this: https://en.m.wikipedia.org/wiki/Universal_approximation_theo...

> what about them

Then you seem to have lost the subject of the thread.

Re: Alice's adventures in a differentiable wonderland

#102

Post by kmmlng » Thu, May 02, 2024, 7:23 AM UTC

jxy wrote: post #23

xanderlewis wrote: post #4

> Stripped of anything else, neural networks are compositions of differentiable primitives I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural…

> > Stripped of anything else, neural networks are compositions of differentiable primitives > I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence. And I hate inaccurate statements like this. It pretends to be rigorous mathematical, but really just propagates erroneous information, and makes the whole article so much m…

Eh, it really doesn't matter much in practice. Additionally, there are many other activation functions without this issue.

Re: Alice's adventures in a differentiable wonderland

#103

Post by srean » Fri, May 03, 2024, 11:38 AM UTC

phkahler wrote: post #95

srean wrote: post #79

Earlier quoted context omitted.

> This is a difference of degree not of kind Nope. Neurons in our brain operate fundamentally differently. They work by transient spikes and information is carried not by the intensity of the spike voltage, but by the frequency of spiking. This is a fundamentally different phenomenon than ANNs where the output (voltage) is a squash transformed aggregated input values (voltages).

>> Neurons in our brain operate fundamentally differently. They work by transient spikes and information is carried not by the intensity of the spike voltage, but by the frequency of spiking. I thought they worked like accumulators where the spike "energy" accumulates until the output "fires". If that's the case then the artificial NNs are still an approximation of that process. I agree that this is a significant dif…

Sandpile mathematics, something that is studied by computer scientists and mathematicians, would be an approximation. Its not so much the level of the spike that matters but how often they spike is what conveys the signal. The temporal behavior is supremely important. It used to be believed that the rate is all that matters, but now, no longer.

There are ANN models that model these spike trains (that's what these 'avalanches' are called), these do work similar to real neurons, but they are not part of the deep neural network popularity [0,1]. Besides, backpropagation is not what goes on in the brain, its known to be biologically infeasible.

So all in all the traditional ANNs are nothing like real neural networks. That's ok, aeroplanes do not fly like birds, but they do still 'fly'.

[0] https://en.wikipedia.org/wiki/Spiking_neural_network

[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9313413/