Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tinker with a Neural Network in Your Browser

playground.tensorflow.org

Post reply on HN
Print view View original

101–110 of 122 posts

Re: Tinker with a Neural Network in Your Browser

#101

Post by daveguy »

DavidSJ wrote: post #98
wjnc wrote: post #87

Earlier quoted context omitted.

Any thoughts on why genetic programming is not 'in fashion'? Does it have anything to do with complexity of the calculations? I can imagine that the advanced models use many, many machines and only deliver results after a large training time. Genetic programming is not feasible then, if you cannot get a quick grasp of the potential results of a model.

If your program is a neural network with N parameters, or a program tree with N nodes, then testing against data takes O(N) time. With evolutionary computation, what you get for your trouble is a single real number -- the loss: how bad it did. With neural networks, backpropagation gives you N real numbers: the gradient of loss with respect to each parameter. Put another way: with evolution you have to stumble around…

> It might take 4.54 billion years.

And then only if you have a system powerful enough to accurately simulate a planet full of molecules.

Although I do think there is a balance between GA and structured NN which will lead to faster and better results than the deep NN alone. We already see some of the best deep NNs incorporating specific structures.

Re: Tinker with a Neural Network in Your Browser

#102

Post by amelius »

erostrate wrote: post #94

The swiss roll problem also illustrates nicely the idea behind deep learning. Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d Then the deep learning guys realized that you don't have to engineer all these…

But how will the number of neurons N grow with the number of turns in the spiral?

If N levels off, then the network has grasped the concept of a spiral and can generalize to arbitrary size.

If N doesn't level off, then the network isn't really learning the general case.

Re: Tinker with a Neural Network in Your Browser

#103

Post by nabla9 »

wjnc wrote: post #87
eggy wrote: post #28

I started reading about ANNs in the 1980s, and had similar confusion to those here, since it was just for fun. I suggest reading a basic book or online information that goes over the basics [1]. I struggled through $200 text books, and jumped from one to the other as an autodidact. I am now studying TWEANNs (Topology and Weight Evolving Artificial Neural Networks), which basically are what you see here with the excep…

Any thoughts on why genetic programming is not 'in fashion'? Does it have anything to do with complexity of the calculations? I can imagine that the advanced models use many, many machines and only deliver results after a large training time. Genetic programming is not feasible then, if you cannot get a quick grasp of the potential results of a model.

Evolutionary algorithms and genetic programming are global optimization technique, basically random search with some memory. It's not "out of fashion" any more than simulated annealing or Monte Carlo methods. They have limited usability, that's all.

Re: Tinker with a Neural Network in Your Browser

#104

Post by iopq »

erostrate wrote: post #94

The swiss roll problem also illustrates nicely the idea behind deep learning. Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d Then the deep learning guys realized that you don't have to engineer all these…

I can reproduce your deep example just fine, but the shallow result needs some luck. At the same time, the shallow result runs faster.

Re: Tinker with a Neural Network in Your Browser

#105

Post by chestervonwinch »

erostrate wrote: post #94

The swiss roll problem also illustrates nicely the idea behind deep learning. Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d Then the deep learning guys realized that you don't have to engineer all these…

Using their network, you are limited to 8 units per layer it seems.

So, I ported their swiss roll dataset to python and threw together a shallow network trainer with theano:

https://gist.github.com/notmatthancock/68d52af2e8cde7fbff1c9...

Then, I trained a shallow network with 36 hidden units (your deep net has 6 units and 6 layers):

http://i.imgur.com/I0pXaTK.png

edit: I forgot to mention that the shallow network above takes only the two coordinates (x1 and x2) as input features.

Re: Tinker with a Neural Network in Your Browser

#106

Post by trgn »

sparky_ wrote: post #62

This is a very cool toy. As someone with no experience in ML, this is an interesting visual approach to the absolute basics. And great for challenging your friends in an epic battle of convergence!

If you like visual demonstrations of ML topics, you may be interested in http://ponder.hepburnave.com. It is an interactive demonstration of a self-organizing map, generating a 2D-map from a spreadsheet with multivariate data. It's an unsupervised learning approach, good for data exploration tasks, less so for classification tasks (/shamelessPlug).

Re: Tinker with a Neural Network in Your Browser

#107

Post by kriro »

erostrate wrote: post #94

The swiss roll problem also illustrates nicely the idea behind deep learning. Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d Then the deep learning guys realized that you don't have to engineer all these…

Along with the images that is a very awesome explanation.

Re: Tinker with a Neural Network in Your Browser

#108

Post by espadrine »

chestervonwinch wrote: post #105
erostrate wrote: post #94

The swiss roll problem also illustrates nicely the idea behind deep learning. Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d Then the deep learning guys realized that you don't have to engineer all these…

Using their network, you are limited to 8 units per layer it seems. So, I ported their swiss roll dataset to python and threw together a shallow network trainer with theano: https://gist.github.com/notmatthancock/68d52af2e8cde7fbff1c9... Then, I trained a shallow network with 36 hidden units (your deep net has 6 units and 6 layers): http://i.imgur.com/I0pXaTK.png edit: I forgot to mention that the shallow network abo…

Just so I understand correctly: your network has 100000 iterations, while the parent's has 1000, but they both only use x / y positions?

It feels like neurons in the first layer are weaker, because all they can do is a linear separation. Given deep networks, I was wondering if adding neurons to the first layer was better than adding them to the last one, and empirically, it feels like it is quite worse. I wonder if there is a theorem around that.

Re: Tinker with a Neural Network in Your Browser

#109

Post by tchow »

erostrate wrote: post #94

The swiss roll problem also illustrates nicely the idea behind deep learning. Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d Then the deep learning guys realized that you don't have to engineer all these…

How do you know to choose 6 hidden layers with 6 neurons each though? Why not 'x' hidden layers with 'j' neurons each? or some other random number?

Also how do you know to choose a ReLu instead of a Tanh activation?

Re: Tinker with a Neural Network in Your Browser

#110

Post by espadrine »

tchow wrote: post #109
erostrate wrote: post #94

The swiss roll problem also illustrates nicely the idea behind deep learning. Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d Then the deep learning guys realized that you don't have to engineer all these…

How do you know to choose 6 hidden layers with 6 neurons each though? Why not 'x' hidden layers with 'j' neurons each? or some other random number? Also how do you know to choose a ReLu instead of a Tanh activation?

ReLu gives good results for deep learning: http://jmlr.org/proceedings/papers/v15/glorot11a/glorot11a.p....

6 layers is the maximum that this demonstration allows, and they kept j small-ish to show that you don't need that many to have good results.

Post reply on HN