Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A visual proof that neural nets can compute any function

neuralnetworksanddeeplearning.com

Post reply on HN
Print view View original

61–67 of 67 posts

Re: A visual proof that neural nets can compute any function

#61

Post by Houshalter »

Retric wrote: post #40
Houshalter wrote: post #39

Earlier quoted context omitted.

The advantage of neural networks is that they can be trained . You can give it a set of inputs and desired outputs, and do gradient descent on it. Neural networks are essentially like trainable digital circuits. The proof of universality shows that neurons can approximate any kind of logic gate (or any input output mapping, like a lookup table.) A lookup table by itself isn't terribly useful, but you can put them in…

A very important caveat is the ability to be trained + universality does not mean they can be trained to fit any function to arbitrary precision in finite time.

Well of course not. If you could fit any neural network to any function quickly, you would have super powers. But in practice, local optima do seem to stop being an issue in big neural networks.

Also this article shows a method of how to construct a lookup table from a neural network in linear time. So in the worst case you can just memorize the input to output table, quickly. In the best case you can fit a simple elegant model, which fits the data perfectly with very few parameters. Given unlimited amounts of time to search the parameter space. Real world NNs are somewhere between these two.

Re: A visual proof that neural nets can compute any function

#62

Post by effie »

forkandwait wrote: post #43

I think an important thing to remember about neural nets is that they are basically a good way to overfit. So, yes, you can approximate any function because you have lots and lots of variables, but you shouldn't fool yourself that you are getting the same information as when you write a deterministic equation with a few variables ("with four parameters I can fit an elephant, with five I can make him wiggle his trunk"…

> I think skilled biological actors use this overfitting to get really good at things without knowing how things actually work.

Perhaps that's the mechanism behind intuition?

Re: A visual proof that neural nets can compute any function

#64

Post by p1esk »

brobdingnagian wrote: post #56
brobdingnagian wrote: post #51

Earlier quoted context omitted.

Perhaps - it's not a bad analogy. But calling myself a neural network is not an analogy - it's literal.

I find it very interesting that I'm being downvoted based on the difference between my self-concept and whoever is doing the downvoting (they don't think they are a neural network). I'm pretty sure that's not what downvotes are supposed to be used for. Perhaps those who are doing the downvoting would be interested in reading the textbook dedicated to the notion that human beings are neural networks, called Computatio…

I think you might be getting downvotes because the neural networks discussed in the article have nothing to do with the neural structures in a human brain, and it appears that you're oblivious to this fact.

Re: A visual proof that neural nets can compute any function

#65

Post by DanBC »

brobdingnagian wrote: post #56
brobdingnagian wrote: post #51

Earlier quoted context omitted.

Perhaps - it's not a bad analogy. But calling myself a neural network is not an analogy - it's literal.

I find it very interesting that I'm being downvoted based on the difference between my self-concept and whoever is doing the downvoting (they don't think they are a neural network). I'm pretty sure that's not what downvotes are supposed to be used for. Perhaps those who are doing the downvoting would be interested in reading the textbook dedicated to the notion that human beings are neural networks, called Computatio…

Stop talking about how downvotes should be used.

It's tedious, and wrong.

You aren't just getting downvotes from people who disagree with you (if that's why you're getting downvotes); you are failing to gain upvotes from anyone who either agrees with you or thinks your downvotes are unfair.

People downvote to disagree - and there's nothing in any site faqs or guidelines to say this is wrong - but those votes are balanced by people upvoting unfairly downvoted posts.

Re: A visual proof that neural nets can compute any function

#66

Post by reagency »

fnbr wrote: post #48
no_gravity wrote: post #17

I wonder how efficiently it can do that compared to other systems. For example a short iterative function like this: function(complex c) complex z=0 int steps=0 while (z Can calculate with extremely high accuracy if a point in the complex plane is in the mandelbrot set or not. I would assume that a NN with the same accuracy would be of enormous size. It would probably have way more neurons then there are atoms in the…

I think you're correct. That's one of the issues with results from theoretical math- the result is true, but it might not be useful. As another commenter in the thread talks about, we can get a nearly identical result by using polynomials of extremely large degree, but that doesn't work well, because of overfitting. We could come up with a polynomial that also calculates with arbitrarily high accuracy if a point is i…

My gut sense is that 10100 degreed wouldn't even be enough to render a pixel accurate mandlebrot set on your phone's screen. So many twisty edges.

Re: A visual proof that neural nets can compute any function

#67

Post by DougMerritt »

otabdeveloper1 wrote: post #27

> Every continuous function in the function space can be represented as a linear combination of basis functions https://en.wikipedia.org/wiki/Basis_function This is basic and obvious math. Does slapping the word 'neural' magically make obvious results 1000% more interesting? Why? Because the word 'neural' carries some of that artificial-intelligence-technology-of-the-future cachet?

> This is basic and obvious math.

Yes. But only once you prove that these are in fact space-spanning basis functions.

Does slapping the world 'basis' magically make results obvious and interesting? Why? Because the word 'basis' carries some of that abstract-mathematics cachet?

Post reply on HN