121–130 of 138 posts
Approximate, not compute. The function also must be continuous. NNs are good for approximation / interpolation / extrapolation, which makes them quite useful for certain domains of problems. But of course, it does not make them a kind of universal computing machine (in the computability sense, like universal Turing machines).
I reckon piecewise continuity is enough.
Earlier quoted context omitted.
When they don't generalize well it usually means you have over trained on your training data. When you take classes on this stuff they have whole sections that talk about trying to detect this and what to do about it e.g. regularization, better models, more training data etc.
What is meant above by "generalisation" is generalisation on out-of-sample data, i.e. not the sample on which you train and test your model (whether you have access to the test set or not). If out-of-sample data has a different distribution than the sampled dataset, then you're SOL, regularisation or not. As a for instance, here's an interesting paper I found out about from HN that describes how image classifiers tra…
Okay, so what? You require more and more neurons (ie. parameters) to approximate your function better and better. You can do the same with piecewise constant (Riemann sums). You can do this with trig functions too (Fourier transform).
"This result tells us that neural networks have a kind of universality."
I don't know what this statement means. What mathematical properties do neural networks have that other functions don't? The ability to approximate continuous functions isn't special. Given 5 points, I can perfectly fit an elephant to your function. And it's not like you are fitting the function with as few parameters as possible.
Earlier quoted context omitted.
You are nitpicking, and completely missing my point. Let me rephrase it. The original article is about neural networks being able to represent any function (for some definition of any). I was just pointing out that there exists much cheaper ways of representing any function. Therefore the article seems very unexciting to me. Btw, you have chose L^2(R) yourself and then used that to show that there are interesting fun…
> You are nitpicking, and completely missing my point. > I was just pointing out that there exists much cheaper ways of representing any function. Therefore the article seems very unexciting to me. To be fair, your original comment didn't really make a point. What kind of cheaper representations do you have in mind? What makes an orthogonal basis of functions too "expensive" a representation for your taste? > I would…
All physics assumes you are dealing with non-pathological functions, except for some really particular cases. You can do nearly everything in Electromagnetism and nearly all Quantum Mechanics with non pathological functions.
Maybe we have a different definition of pathological, I am using it in the way a physicist would use (i.e. continuous, continuous derivatives, so on)
Approximate, not compute. The function also must be continuous. NNs are good for approximation / interpolation / extrapolation, which makes them quite useful for certain domains of problems. But of course, it does not make them a kind of universal computing machine (in the computability sense, like universal Turing machines).
Ok we'll give you some approximation and some continuousness in the title above.
Earlier quoted context omitted.
> You are nitpicking, and completely missing my point. > I was just pointing out that there exists much cheaper ways of representing any function. Therefore the article seems very unexciting to me. To be fair, your original comment didn't really make a point. What kind of cheaper representations do you have in mind? What makes an orthogonal basis of functions too "expensive" a representation for your taste? > I would…
Most functions are pathological? Are you serious? All physics assumes you are dealing with non-pathological functions, except for some really particular cases. You can do nearly everything in Electromagnetism and nearly all Quantum Mechanics with non pathological functions. Maybe we have a different definition of pathological, I am using it in the way a physicist would use (i.e. continuous, continuous derivatives, so…
The kind of pathological function that I'm referring to is neither of these. For example, what does the manifold of all 1 second clips of the word "the" look like? If the clip is sampled at 60 Hz, each clip is already in 60d space. I'm inclined to think that it's some unimaginably complicated manifold that would likely fall into the category of "edge cases", which previous commenters have mentioned and it sounds like you're discounting as nitpicking.
I don't know if this aligns with what you mean by a pathological function, and I'm happy to continue having this discussion with a more concrete example of what you mean. :)
Of course they can, because an NN can be any function. If we can pick tanh as the "activation" then we can as easily pick arctan as the activation ans say our NN computes arctan. What an achievement! A better question is whether conv+relu based NNs can approximate any function. But that's most likely false because there are many weird functions that are impossible to compute, not even approximate (I'm talking about t…
This is a non sequitur in this context. The universality described here depends only on changing connection weights, not the neuronal activation functions. An important caveat is the approximated function must be continuous, but that covers a very large family.