Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What does a neural network actually do?

moalquraishi.wordpress.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: What does a neural network actually do?

#31

Post by sergiosgc »

robert_tweed wrote: post #22

This is a pretty good article, but I'm seeing a lot of confusion in this thread because the article is maybe one step ahead of the basic intuition needed to understand why ANNs are not magical and are not artificial intelligence (at least not feed-forward networks). Perhaps a simpler way to look at it is to understand that a feed-forward ANN is basically just a really fancy transformation matrix. OK, so unless you kn…

> The only difference between a transformation matrix and a neural network is that a neural network has at least two layers. In other words, it is two (or more) transformation matrices bolted together. For reasons that are a bit too complex to get into here, allows an NN to perform more complex transformations than a single matrix can. In fact, it turns out that an arbitrarily large NN can perform any polynomial-based transformation on the data.

Nice explanation. I need one clarification, though. Isn't matrix multiplication associative? Isn't thus any transformation defined by two matrices representable by a single matrix that is the product of the two matrices?

I am probably misunderstanding how NNs bolt matrices together.

Re: What does a neural network actually do?

#32

Post by tfgg »

sergiosgc wrote: post #31
robert_tweed wrote: post #22

This is a pretty good article, but I'm seeing a lot of confusion in this thread because the article is maybe one step ahead of the basic intuition needed to understand why ANNs are not magical and are not artificial intelligence (at least not feed-forward networks). Perhaps a simpler way to look at it is to understand that a feed-forward ANN is basically just a really fancy transformation matrix. OK, so unless you kn…

> The only difference between a transformation matrix and a neural network is that a neural network has at least two layers. In other words, it is two (or more) transformation matrices bolted together. For reasons that are a bit too complex to get into here, allows an NN to perform more complex transformations than a single matrix can. In fact, it turns out that an arbitrarily large NN can perform any polynomial-base…

You apply a non-linear function (usually some sigmoid) on the output vector after each matrix product. Otherwise, you'd be correct and any multi-layer ANN could be expressed as a single layer network.

Re: What does a neural network actually do?

#33

Post by lloeki »

robert_tweed wrote: post #22

This is a pretty good article, but I'm seeing a lot of confusion in this thread because the article is maybe one step ahead of the basic intuition needed to understand why ANNs are not magical and are not artificial intelligence (at least not feed-forward networks). Perhaps a simpler way to look at it is to understand that a feed-forward ANN is basically just a really fancy transformation matrix. OK, so unless you kn…

So, IIUC creating a NN basically follows this process:

- define an input vector space (i.e choose dimensions you want to operate on with input data)

- define your categories in another space (or another basis in the same space?)

- set up a transformation pipeline between the two spaces (with at least two stages)

- devise an algorithm that takes categorised elements and produces new transformation matrices

- train the NN (i.e feed input and categorise the result so that through some algorithm the transformation matrices converge)

Re: What does a neural network actually do?

#34

Post by dwiel »

sergiosgc wrote: post #31
robert_tweed wrote: post #22

This is a pretty good article, but I'm seeing a lot of confusion in this thread because the article is maybe one step ahead of the basic intuition needed to understand why ANNs are not magical and are not artificial intelligence (at least not feed-forward networks). Perhaps a simpler way to look at it is to understand that a feed-forward ANN is basically just a really fancy transformation matrix. OK, so unless you kn…

> The only difference between a transformation matrix and a neural network is that a neural network has at least two layers. In other words, it is two (or more) transformation matrices bolted together. For reasons that are a bit too complex to get into here, allows an NN to perform more complex transformations than a single matrix can. In fact, it turns out that an arbitrarily large NN can perform any polynomial-base…

Interesting to note though that even with a linear network that can be represented by a single matrix, it can be faster, easier and converge to better results with multiple layers because the different gradient and parameter space that is presented to the optimization algorithm.

Re: What does a neural network actually do?

#35

Post by chestervonwinch »

nanidin wrote: post #29
nullc wrote: post #28

Another limit they don't address is that the training normally used is purely local— just a gradient descent. So even when the network can model your function well, there is no guarantee that it will find the solution. For me ANN's always seem to get stuck on not very helpful local minima— they're not one of the first tools in my bags of tricks by far. Often I associate them as being the sort of thing that someone wh…

Well, you could use an EA to take a stab at finding better minima :) And correct me if I'm wrong, but isn't the cost function for a feed forward neural networks that uses a sigmoid activation function convex wrt the parameters being trained, i.e. gradient descent is guaranteed to find the global minimum when small enough of a step size is used?

Mostly, no. Hidden units introduce non-convexity to the cost. How bout a simple counter-example?

Take a simple classifier network with one input, one hidden unit and one output and no biases. To make things even simpler, tie the two weights, i.e. make the first weight equal to the second. Now, mathematically the output of the network can be written: z=f(w * f(w * x)) where f() is the sigmoid.

Next, consider a dataset with two items: [(x_1, y_1), (x_2, y_2)] where x_i is the input and y_i is the class label, 0 or 1. Take as values: [(0.9, 1), (0.1,0)]. The cost function (loglikelihood in this case) is:

L(w) = sum_i { y_i * log( f(w * f(w * x_i)) ) + (1-y_i) * log( 1-f(w * f(w * x_i)) ) }

or

L(w) = log( f(w * f(w * 0.9)) ) + log( 1-f(w * f(w * 0.1)) )

Plot that last guy replacing f with the sigmoid, and you'll see the result is non-convex - there's a kink near zero.

Re: What does a neural network actually do?

#36

Post by joe_the_user »

sergiosgc wrote: post #31
robert_tweed wrote: post #22

This is a pretty good article, but I'm seeing a lot of confusion in this thread because the article is maybe one step ahead of the basic intuition needed to understand why ANNs are not magical and are not artificial intelligence (at least not feed-forward networks). Perhaps a simpler way to look at it is to understand that a feed-forward ANN is basically just a really fancy transformation matrix. OK, so unless you kn…

> The only difference between a transformation matrix and a neural network is that a neural network has at least two layers. In other words, it is two (or more) transformation matrices bolted together. For reasons that are a bit too complex to get into here, allows an NN to perform more complex transformations than a single matrix can. In fact, it turns out that an arbitrarily large NN can perform any polynomial-base…

Well,

I suspect some kind of transformation could be used to make a two level NN into a one level one. The thing is the resulting one level network might be more complex and less useful than the original two level network. Still, I think this does illustrate the limitations of multilevel networks.

Another way to see this is to notice that NNs and SVMs[1] are (approximately or exactly) equivalent [2] because they both involve the fairly simple linear and non-linear transformations we've been looking at.

[1] http://en.wikipedia.org/wiki/Support_vector_machine [2] http://www.staff.ncl.ac.uk/peter.andras/PAnpl2002.pdf

Re: What does a neural network actually do?

#38

Post by sergiosgc »

tfgg wrote: post #32
sergiosgc wrote: post #31

Earlier quoted context omitted.

> The only difference between a transformation matrix and a neural network is that a neural network has at least two layers. In other words, it is two (or more) transformation matrices bolted together. For reasons that are a bit too complex to get into here, allows an NN to perform more complex transformations than a single matrix can. In fact, it turns out that an arbitrarily large NN can perform any polynomial-base…

You apply a non-linear function (usually some sigmoid) on the output vector after each matrix product. Otherwise, you'd be correct and any multi-layer ANN could be expressed as a single layer network.

Thanks. It makes sense. The sigmoid is the activation function of the output "neuron". Unfortunately, matrix algebra here is not as useful as in computer graphics.

Re: What does a neural network actually do?

#39

Post by tfgg »

sergiosgc wrote: post #38
tfgg wrote: post #32

Earlier quoted context omitted.

You apply a non-linear function (usually some sigmoid) on the output vector after each matrix product. Otherwise, you'd be correct and any multi-layer ANN could be expressed as a single layer network.

Thanks. It makes sense. The sigmoid is the activation function of the output "neuron". Unfortunately, matrix algebra here is not as useful as in computer graphics.

No problem. Actually, I personally found that a pretty intuitive understanding of linear algebra & vector calculus makes quite a lot of ML straight forward to approach geometrically.
Post reply on HN