I skimmed it, but at a glance, isn't this almost identical to the Taylor Series? https://en.wikipedia.org/wiki/Taylor_series
Well, in the sense that polynomials are universal approximators too (on a compact domain). See this comment:
A visual proof that neural nets can approximate any function
81–90 of 138 posts
Re: A visual proof that neural nets can approximate any function
#82Re: A visual proof that neural nets can approximate any function
#83Earlier quoted context omitted.
A classic demonstration of similar effect - any set of N data points in a time series, e.g. (t,f(t)), can be fit by a N-1 order polynomial to pass through each point. So fit a high order poly to a set of points sampled (esp. with a little noise) from a low order poly. You'll get crazy oscillations, and outside the sampling area it will likely diverge fast. Now add a smoothness term and crank it up until you get more…
One of my labs some students curve fitted a sixth order poly onto five data points they collected. The process being measured was y = something something minus ln(x). It fit all five points exactly and smoothly and wildly diverged on either side. The professor was really amused.
It’s called
https://en.m.wikipedia.org/wiki/Runge's_phenomenon
and can be mitigated eg by non-uniform interpolation grids like chebychev nodes.
Re: A visual proof that neural nets can approximate any function
#84Approximate, not compute. The function also must be continuous. NNs are good for approximation / interpolation / extrapolation, which makes them quite useful for certain domains of problems. But of course, it does not make them a kind of universal computing machine (in the computability sense, like universal Turing machines).
> NNs are good for approximation / interpolation / extrapolation Extrapolation? I was under the impression that generalizability of NNs beyond the training data was one of the major problems faced by NNs.
You're correct. As long as your test set remains within the training distribution, you can expect the NN to be well behaved. However, its behavior is undefined for testing data out of training distribution. There is a lot of work on detecting out of distribution inputs, regularizing NNs to follow a simple prior, etc. but the core problem remains because NNs learn from data. Extrapolation is something that symbolic systems do well, and NNs do not.
Re: A visual proof that neural nets can approximate any function
#85Approximate, not compute. The function also must be continuous. NNs are good for approximation / interpolation / extrapolation, which makes them quite useful for certain domains of problems. But of course, it does not make them a kind of universal computing machine (in the computability sense, like universal Turing machines).
The line between approximate and compute is a blurry one: some may say if the polynomial approximation depends on log of the error, meaning you can exponentially get better with each step, computation and approximation are the same.
Re: A visual proof that neural nets can approximate any function
#86Approximate, not compute. The function also must be continuous. NNs are good for approximation / interpolation / extrapolation, which makes them quite useful for certain domains of problems. But of course, it does not make them a kind of universal computing machine (in the computability sense, like universal Turing machines).
The line between approximate and compute is a blurry one: some may say if the polynomial approximation depends on log of the error, meaning you can exponentially get better with each step, computation and approximation are the same.
I think it is important in the context of what people are trying to use NNs for. Most of the time it is some type of classification, in which case there is always a chance it will be completely wrong in the sense that anything other than the expected output is not correct in any way.
Which leads into unknowingly building NNs that are actually building classification networks and not realizing approximations might not fit into your model.
Re: A visual proof that neural nets can approximate any function
#87Earlier quoted context omitted.
> NNs are good for approximation / interpolation / extrapolation Extrapolation? I was under the impression that generalizability of NNs beyond the training data was one of the major problems faced by NNs.
When they don't generalize well it usually means you have over trained on your training data. When you take classes on this stuff they have whole sections that talk about trying to detect this and what to do about it e.g. regularization, better models, more training data etc.
What you list remains insufficient to tackle the difficulty of extrapolation. Extrapolation of the kind we're able to do with Physics theories is difficult in the general case for all methods, not just deep learning. With even the relevant variables subject to change, things like distribution shift and non-stationarity are but the tip of the ice-berg.
For neural networks, if you take something basic like sorting a list or multiplying two decimal numbers, the further you are from range the models were trained on, the worse they will do (yes, transformers too). Only exception I can think of are carefully trained Neural GPUs, which will quickly struggle to be 100% correct as you depart simple tasks. While consuming a great deal of computational resources. Program synthesis is the general area, with no approach clearly dominant in the same way deep learning has dominated machine learning.
Re: A visual proof that neural nets can approximate any function
#88I skimmed it, but at a glance, isn't this almost identical to the Taylor Series? https://en.wikipedia.org/wiki/Taylor_series
Well, in the sense that polynomials are universal approximators too (on a compact domain). See this comment: https://news.ycombinator.com/item?id=19709834
The Stone-Weierstrass theorem is not related to Taylor series, in fact the target function need not have a derivative, let alone have all higher-order derivatives, a prerequisite for the Taylor series to even exist.
Re: A visual proof that neural nets can approximate any function
#89I skimmed it, but at a glance, isn't this almost identical to the Taylor Series? https://en.wikipedia.org/wiki/Taylor_series
No.