A visual proof that neural nets can compute any function • Hacker News Board

Re: A visual proof that neural nets can compute any function

#81

Post by Houshalter » Thu, Sep 04, 2014, 10:31 PM UTC

mtdewcmu wrote: post #80

Houshalter wrote: post #69

Earlier quoted context omitted.

It can learn any input to output mapping. What more do you want?

The article doesn't say anything about whether it can learn it, just that it can represent it.

The proof is constructive, so actually you can. Just create a few neurons for the "tower function" and the correct output weight at every point you want to map.

If you mean learn functions efficiently with few parameters, or in ways that generalize well, the article makes no claims about that at all. There are two "no free lunch" theorems. One of which says that it's impossible to guarantee any method will generalize well on all problems, and the other that says it's impossible to guarantee any optimization method will work well on all problems. You just have to make strong assumptions like "my function can be modeled by a neural net" or "the error function is convex."

However neural networks are agnostic to the optimization algorithm you use to set their weights. There are many different ones.

Re: A visual proof that neural nets can compute any function

#82

Post by jokoon » Fri, Sep 05, 2014, 12:19 PM UTC

j2kun wrote: post #10

jokoon wrote: post #8

who's to doubt neural networks are completely awesome ? Any more news about those chips that were optimized for neural networks ? Was it IBM or Samsung ?

We have very few theoretical results about neural networks. So not completely awesome.

I mean there is a big future for neural networks

Re: A visual proof that neural nets can compute any function

#83

Post by j2kun » Fri, Sep 05, 2014, 1:27 PM UTC

jokoon wrote: post #82

j2kun wrote: post #10

Earlier quoted context omitted.

We have very few theoretical results about neural networks. So not completely awesome.

I mean there is a big future for neural networks

Well you asked a question and I answered it honestly :)

Re: A visual proof that neural nets can compute any function

#84

Post by j2kun » Fri, Sep 05, 2014, 1:30 PM UTC

mtdewcmu wrote: post #51

Tloewald wrote: post #48

Can't the same be said for Fourier series, which make no claims to be some kind of AI? And likewise humble polynomials: http://en.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_theor...

Yes (I am not expert in neural nets, but that appears to be exactly what this is saying). If you look at what goes into a neural net and compare it to what goes into a Fourier transform, it should be obvious that neural nets have even more than they actually need to do this task.

This statement doesn't make sense to me. A neural network literally can't produce anything besides a continuous function, and the universality theorem says that there is no continuous function they can't (approximately) produce.

So what could you possibly mean when you say neural networks have "more" than they need to do something which characterizes exactly what they can and can't do?

Re: A visual proof that neural nets can compute any function

#85

Post by jokoon » Sun, Sep 07, 2014, 12:01 PM UTC

j2kun wrote: post #83

jokoon wrote: post #82

Earlier quoted context omitted.

I mean there is a big future for neural networks

Well you asked a question and I answered it honestly :)

> So not completely awesome.

I don't think I understand your point. By awesome I meant it's a very exciting/interesting field of research. I don't mean to call thermodynamics awesome because it powers cars or because it's a well research field.

Trying to understand how the brain works or making a computer that works in a similar way is awesome to me. If there are still computational limitations that makes it not practical, that's still an awesomely interesting subject.

Re: A visual proof that neural nets can compute any function

#86

Post by mtdewcmu » Mon, Sep 08, 2014, 7:45 AM UTC

j2kun wrote: post #84

mtdewcmu wrote: post #51

Earlier quoted context omitted.

Yes (I am not expert in neural nets, but that appears to be exactly what this is saying). If you look at what goes into a neural net and compare it to what goes into a Fourier transform, it should be obvious that neural nets have even more than they actually need to do this task.

This statement doesn't make sense to me. A neural network literally can't produce anything besides a continuous function, and the universality theorem says that there is no continuous function they can't (approximately) produce. So what could you possibly mean when you say neural networks have "more" than they need to do something which characterizes exactly what they can and can't do?

There are more coefficients than necessary. The FT has the minimum; it's bijective.