Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A visual proof that neural nets can approximate any function

neuralnetworksanddeeplearning.com

Post reply on HN
Print view View original

101–110 of 138 posts

Re: A visual proof that neural nets can approximate any function

#101

Post by [deleted] »

johnwatson11218 wrote: post #52
electricslpnsld wrote: post #40

Earlier quoted context omitted.

> NNs are good for approximation / interpolation / extrapolation Extrapolation? I was under the impression that generalizability of NNs beyond the training data was one of the major problems faced by NNs.

When they don't generalize well it usually means you have over trained on your training data. When you take classes on this stuff they have whole sections that talk about trying to detect this and what to do about it e.g. regularization, better models, more training data etc.

[deleted]

Re: A visual proof that neural nets can approximate any function

#102

Post by mturmon »

xtacy wrote: post #53
13415 wrote: post #47

Earlier quoted context omitted.

Wait a minute, isn't it also the case that according to the Weierstrass approximation theorem any continuous function on a closed interval can be approximated by a polynomial function? And isn't that kind of pointless for practical applications because we also need to avoid overfitting? To clarify, I'm not trying to make a snippy remark, I just happened to have used polynomial curve fitting before and looked up the W…

Yes, the claims are pretty much in the same spirit. Although the first (Weierstrass's) theorem [1] was stated for real-valued functions in a 1-D closed interval [a, b], Stone-Weirstrass is a generalisation of the above theorem [2] that's applicable in more general scenarios. Here is the formal statement: - [1] http://mathworld.wolfram.com/WeierstrassApproximationTheorem... - [2] http://mathworld.wolfram.com/Stone-Wei…

Yes. A NN consisting of, for example, sigmoids will form an “algebra” that separates points in the sense of the Stone Weierstrass theorem, and the NN approximation result will follow.

All that’s really needed is that the limit of the NN basis function is different at plus versus minus infinity on the real line. This will give you the “separates points “ property.

Re: A visual proof that neural nets can approximate any function

#103

Post by YeGoblynQueenne »

johnwatson11218 wrote: post #52
electricslpnsld wrote: post #40

Earlier quoted context omitted.

> NNs are good for approximation / interpolation / extrapolation Extrapolation? I was under the impression that generalizability of NNs beyond the training data was one of the major problems faced by NNs.

When they don't generalize well it usually means you have over trained on your training data. When you take classes on this stuff they have whole sections that talk about trying to detect this and what to do about it e.g. regularization, better models, more training data etc.

What is meant above by "generalisation" is generalisation on out-of-sample data, i.e. not the sample on which you train and test your model (whether you have access to the test set or not). If out-of-sample data has a different distribution than the sampled dataset, then you're SOL, regularisation or not.

As a for instance, here's an interesting paper I found out about from HN that describes how image classifiers trained on one standard dataset (ImageNet etc) do much worse on other standard datasets and how it's even possible to identify the dataset a classifier was trained on:

Unbiased look at dataset bias

https://people.csail.mit.edu/torralba/publications/datasets_...

Re: A visual proof that neural nets can approximate any function

#104

Post by otabdeveloper2 »

> neural nets can approximate any continuous function

So-called "neural nets" are just logistic regression with a fancy name. Seems "deep learning" is just a wavelet transform with a sigmoid basis. (So, boring math stuff we already knew forever, plus marketing mumbo-jumbo.)

Re: A visual proof that neural nets can approximate any function

#105

Post by Gravityloss »

mkl wrote: post #77

I think you can explain it even more clearly with smooth relative shifts, rather than sharp bump functions. I made a quick demo: https://www.desmos.com/calculator/rfaqogkbmy Drag the sliders for w and n to change how step-like the sigmoids are and how many are combined. The purple lines are the sigmoids, relative changes at each (regularly spaced) position, which are added together to make the blue approximation to t…

This deserves its own submission. Maybe the number of neurons slider could be logarithmic?

Re: A visual proof that neural nets can approximate any function

#106

Post by soVeryTired »

otabdeveloper2 wrote: post #104

> neural nets can approximate any continuous function So-called "neural nets" are just logistic regression with a fancy name. Seems "deep learning" is just a wavelet transform with a sigmoid basis. (So, boring math stuff we already knew forever, plus marketing mumbo-jumbo.)

> Seems "deep learning" is just a wavelet transform with a sigmoid basis

Well... more like an iterated transform.

You're right, most of the mathematics is very old indeed. What changed was the hardware (parallelism), software (easy-to-use autodiff packages) and the availability of data.

There's a lot of hype in the field, but some of that hype is deserved. Computer vision was practically in crisis in the late 00's and early 10's. No significant progress was being made on problems, and there were few strategic directions to move in that hadn't been done to death already. Then smash: along comes deep learning, which changed everything.

Re: A visual proof that neural nets can approximate any function

#107

Post by pauljurczak »

bhl wrote: post #85
a-nikolaev wrote: post #31

Approximate, not compute. The function also must be continuous. NNs are good for approximation / interpolation / extrapolation, which makes them quite useful for certain domains of problems. But of course, it does not make them a kind of universal computing machine (in the computability sense, like universal Turing machines).

The line between approximate and compute is a blurry one: some may say if the polynomial approximation depends on log of the error, meaning you can exponentially get better with each step, computation and approximation are the same.

My elementary school teacher would disagree: 2+2=4 not somewhere around 3.997.

Re: A visual proof that neural nets can approximate any function

#108

Post by soVeryTired »

throwawaymath wrote: post #61
braised_babbage wrote: post #30

Earlier quoted context omitted.

One of the things about continuous functions is that they aren't necessarily differentiable, cf. https://en.wikipedia.org/wiki/Weierstrass_function

True, but that was kind of an afterthought of my point. All continuous functions still comprise a vector space.

I don't see how that helps?

> A neural network computes linear transformations

They compute a nonlinear function of an affine transform. Neural networks are nonlinear functions.

Re: A visual proof that neural nets can approximate any function

#109

Post by emilfihlman »

codesternews wrote: post #4

Any deeplearning expert here. Why Neural network can't compute a linear function Celsius to Fahrenheit 100% accurately. Is it data or is it something can be optimised. ``` celsius_q = np.array([-40, -10, 0, 8, 15, 22, 38], dtype=float) fahrenheit_a = np.array([-40, 14, 32, 46, 59, 72, 100], dtype=float) for i,c in enumerate(celsius_q): print("{} degrees Celsius = {} degrees Fahrenheit".format(c, fahrenheit_a[i])) l0…

Your activation function needs to be linear to be able to achieve fully linear results.

However, this kills the neural network as a general purpose computation device.

Re: A visual proof that neural nets can approximate any function

#110

Post by pauljurczak »

tsbinz wrote: post #11
hcs wrote: post #3

Earlier quoted context omitted.

> First, this doesn't mean that a network can be used to exactly compute any function. Rather, we can get an approximation that is as good as we want. By increasing the number of hidden neurons we can improve the approximation.

Also, the article doesn't mention a crucial part of the universal approximation theorem: It is about functions on compact subsets of R^n, so it doesn't say anything about functions that take the whole of R as input.

They can't approximate functions on unbound domain in general case.
Post reply on HN