Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What does a neural network actually do?

moalquraishi.wordpress.com

Post reply on HN
Print view View original

11–20 of 39 posts

Re: What does a neural network actually do?

#11

Post by im3w1l »

billderose wrote: post #9
csense wrote: post #3

I've found that, in practice, traditional neural networks tend to be prone to overfitting and are finicky about their parameters (in particular the topology and number of nodes you choose). I use the word "traditional" to describe the NN architecture discussed in the article. Recent NN research has been promising [1], but this article strictly discusses traditional NN's. I don't really have much experience with the n…

Hinton's DropOut [1] and Wan's DropConnect [2] have ameliorated some of the overfitting issues present in traditional NN's. In fact, DropConnect in conjunction with deep learning are responsible for new records being set on classical datasets such as MNIST. [1] http://arxiv.org/pdf/1207.0580.pdf [2] http://cs.nyu.edu/~wanli/dropc/

It's pretty funny, I saw DropConnect described in a stackoverflow answer that predated the paper you reference. It was an incorrect answer on how to do dropout. I shall try to find it tomorrow.

Re: What does a neural network actually do?

#12

Post by ShardPhoenix »

dlsym wrote: post #8

"What does a neural network actually do?" This is a fundamental Question: Can we really say and predict what a neural network does? Contrary to an engineered / constructed algorithm a neural network is 'trained'. Whenever we will present a 'known' input pattern it will reposnd with a 'learned' response. This however introduces interesting problems: How can we _debug_ a neural network? How can we debug a correlation?…

I can't find an example on Google right now, but I've seen demonstrations that it's possible to visualize the intermediate layers of a neural network - for example you can see how an image recognition network is first breaking down an image into horizontal and vertical lines, then combining those into more complex shapes, etc.

Re: What does a neural network actually do?

#14

Post by agibsonccc »

ShardPhoenix wrote: post #12
dlsym wrote: post #8

"What does a neural network actually do?" This is a fundamental Question: Can we really say and predict what a neural network does? Contrary to an engineered / constructed algorithm a neural network is 'trained'. Whenever we will present a 'known' input pattern it will reposnd with a 'learned' response. This however introduces interesting problems: How can we _debug_ a neural network? How can we debug a correlation?…

I can't find an example on Google right now, but I've seen demonstrations that it's possible to visualize the intermediate layers of a neural network - for example you can see how an image recognition network is first breaking down an image into horizontal and vertical lines, then combining those into more complex shapes, etc.

These are called filters. This is from my deep learning framework: Debugging a net visually: http://deeplearning4j.org/debug.html

An example of doing facial reconstruction: http://deeplearning4j.org/facial-reconstruction-tutorial.htm...

Re: What does a neural network actually do?

#15

Post by agibsonccc »

oldspiceman wrote: post #13

A less mathy explanation with some real examples: http://neuralnetworksanddeeplearning.com/chap1.html Coding a digit recognizer using a neural network is an extremely rewarding exercise and there's a lot of help on the web to get you started.

This is a great example of a hello world application. Keep in mind there are several kinds of neural nets that allow you to do this. This includes convolutional RBMs (recognizes parts of an image) and normal RBMs (learns everything at once)

Re: What does a neural network actually do?

#16

Post by agibsonccc »

billderose wrote: post #9
csense wrote: post #3

I've found that, in practice, traditional neural networks tend to be prone to overfitting and are finicky about their parameters (in particular the topology and number of nodes you choose). I use the word "traditional" to describe the NN architecture discussed in the article. Recent NN research has been promising [1], but this article strictly discusses traditional NN's. I don't really have much experience with the n…

Hinton's DropOut [1] and Wan's DropConnect [2] have ameliorated some of the overfitting issues present in traditional NN's. In fact, DropConnect in conjunction with deep learning are responsible for new records being set on classical datasets such as MNIST. [1] http://arxiv.org/pdf/1207.0580.pdf [2] http://cs.nyu.edu/~wanli/dropc/

Dropout is actually a knob on any neural network. These are used in image recognition as well as text and other areas.

The fuzzing creates a very similar effect to convolutional nets where it can learn different poses of an image.

Re: What does a neural network actually do?

#17

Post by joe_the_user »

dlsym wrote: post #8

"What does a neural network actually do?" This is a fundamental Question: Can we really say and predict what a neural network does? Contrary to an engineered / constructed algorithm a neural network is 'trained'. Whenever we will present a 'known' input pattern it will reposnd with a 'learned' response. This however introduces interesting problems: How can we _debug_ a neural network? How can we debug a correlation?…

I think you have hit upon a problem of present day AI.

Neural networks, Support Vector Machines, Hidden Markov Models and other stuff (Markov Networks, etc) do something like linear regression on some huge space - they draw a curve/plane between groups of things on this feature space. The tendency is for this division to make sense and to correspond to our common sense categorization of these things.

The problem is that once that happens, you can't really reason about the division you've drawn. It's just there. You can tweak for various purposes but that's a manual process.

You categorize animals by shape or particular adaptation or by genetic makeup. You can teach one of these algorithm each of these categorizations. But you can't do something like have the thing categorize for one purpose and then tell it to "change it's outlook" and categorize for a different purpose.

In this sense, despite seeming impressive, the products of these processes are dead-ends that we can't reason about, that lack the flexible intelligence of a human being.

Re: What does a neural network actually do?

#18

Post by joe_the_user »

ShardPhoenix wrote: post #12
dlsym wrote: post #8

"What does a neural network actually do?" This is a fundamental Question: Can we really say and predict what a neural network does? Contrary to an engineered / constructed algorithm a neural network is 'trained'. Whenever we will present a 'known' input pattern it will reposnd with a 'learned' response. This however introduces interesting problems: How can we _debug_ a neural network? How can we debug a correlation?…

I can't find an example on Google right now, but I've seen demonstrations that it's possible to visualize the intermediate layers of a neural network - for example you can see how an image recognition network is first breaking down an image into horizontal and vertical lines, then combining those into more complex shapes, etc.

But visualizing is quite a ways from debugging.

To debug a program you actually verify that it's logic is correct (at least the good kind of debugging).

Consider a spectrum:

1. Natural language - we humans combine fragments of natural language easily and on an ad-hoc basis. We can get a fairly amount of use from reusing Shakespeare quotes and neologisms while spending rather little effort.

2. Trained programmers can reuse and combine general purpose libraries - with difficulty and often after considerable debugging.

3. AI algorithms like Neural Networks. These are just plopped in and tweaked, not combining seems possible.

It seem like "intelligent behavior" should be going more towards #1 but the process of Machine Learning seems to move things more towards #3. The "learn once, understand never" approach means that for each significant case, you'll need to do a re-tweaking and re-learning. The potential to get harder rather than easier over time might well be there.

Re: What does a neural network actually do?

#19

Post by aidos »

ShardPhoenix wrote: post #12
dlsym wrote: post #8

"What does a neural network actually do?" This is a fundamental Question: Can we really say and predict what a neural network does? Contrary to an engineered / constructed algorithm a neural network is 'trained'. Whenever we will present a 'known' input pattern it will reposnd with a 'learned' response. This however introduces interesting problems: How can we _debug_ a neural network? How can we debug a correlation?…

I can't find an example on Google right now, but I've seen demonstrations that it's possible to visualize the intermediate layers of a neural network - for example you can see how an image recognition network is first breaking down an image into horizontal and vertical lines, then combining those into more complex shapes, etc.

There's an interesting example in one of the coursera courses (Neural Networks for Machine Learning) - you just need to watch through the intro video to see it in action.

https://www.coursera.org/course/neuralnets

Re: What does a neural network actually do?

#20

Post by judk »

joe_the_user wrote: post #17
dlsym wrote: post #8

"What does a neural network actually do?" This is a fundamental Question: Can we really say and predict what a neural network does? Contrary to an engineered / constructed algorithm a neural network is 'trained'. Whenever we will present a 'known' input pattern it will reposnd with a 'learned' response. This however introduces interesting problems: How can we _debug_ a neural network? How can we debug a correlation?…

I think you have hit upon a problem of present day AI. Neural networks, Support Vector Machines, Hidden Markov Models and other stuff (Markov Networks, etc) do something like linear regression on some huge space - they draw a curve/plane between groups of things on this feature space. The tendency is for this division to make sense and to correspond to our common sense categorization of these things. The problem is t…

Your criticism is fair, but you fail to explain how an NN or SVM is any worse than how a human mind actually operates.

In other words, the incomprehensibility of a modern AI model is not a failing of AI, it is a failing of (AI) psychology and (AI) neuroscience.

The artificially constructed intelligence works whether or not we understand how. The frontier of AI science is now open to AI Psychology. Psychologists and Neuroscientists will replace the data scientists! Such fun!

Post reply on HN