Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tinker with a Neural Network in Your Browser

playground.tensorflow.org

Post reply on HN
Print view View original

91–100 of 122 posts

Re: Tinker with a Neural Network in Your Browser

#91

Post by argonaut »

wjnc wrote: post #87
eggy wrote: post #28

I started reading about ANNs in the 1980s, and had similar confusion to those here, since it was just for fun. I suggest reading a basic book or online information that goes over the basics [1]. I struggled through $200 text books, and jumped from one to the other as an autodidact. I am now studying TWEANNs (Topology and Weight Evolving Artificial Neural Networks), which basically are what you see here with the excep…

Any thoughts on why genetic programming is not 'in fashion'? Does it have anything to do with complexity of the calculations? I can imagine that the advanced models use many, many machines and only deliver results after a large training time. Genetic programming is not feasible then, if you cannot get a quick grasp of the potential results of a model.

At least for deep learning, most deep learning models take more than a week to train, often on multiple GPUs. Some of the extremely deep, huge dataset models can take multiple weeks on multiple GPUs. Google trained AlphaGo's nets for months (on god knows how many GPU/CPUs). Suffice to say, people don't even bother touching most hyperparameters, let alone trying to do something more exhaustive.

Re: Tinker with a Neural Network in Your Browser

#92

Post by [deleted] »

sparky_ wrote: post #61
nl wrote: post #56

This is great, but I think they should make it clear that this isn't using TensorFlow. From the title and domain I though they either had ported TF to Javascript(!) or we connecting to a server.

Wait - what it is using, then? I had assumed it was TF under Emscripten or similar.

[deleted]

Re: Tinker with a Neural Network in Your Browser

#94

Post by erostrate »

The swiss roll problem also illustrates nicely the idea behind deep learning.

Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d

Then the deep learning guys realized that you don't have to engineer all these extra features, you can just use basic features x_1, x_2 and let the network learn more complicated transformations in subsequent layers. So they would use a deep network with only x_1, x_2 as inputs: http://imgur.com/XBRjROP

Both these approaches work here (loss < 0.01). The difference is that for the first one you have to manually choose the extra features sin(x_1), x_1^2, ... for each problem. And the more complicated the problem the harder it is to design good features. People in the computer vision community spent years and years trying to design good features for e.g. object recognition. But finally some people realized that deep networks could learn these features themselves. And that's the main idea in deep learning.

Re: Tinker with a Neural Network in Your Browser

#96

Post by raverbashing »

erostrate wrote: post #94

The swiss roll problem also illustrates nicely the idea behind deep learning. Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d Then the deep learning guys realized that you don't have to engineer all these…

This is a very good explanation, thanks (even though I knew some of it already)

I tried the swiss roll with a shallow network on the demo (and the results are not excellent, but it matches)

Re: Tinker with a Neural Network in Your Browser

#97

Post by samscully »

ylem wrote: post #50
argonaut wrote: post #40

Earlier quoted context omitted.

As far as the recent deep learning boom is concerned, genetic programming is really out of favor. I don't really see it in any of the deep learning (or even machine learning, for that matter) literature/successes/research groups. "Neural networks" are a really really overloaded term. A ton of stuff referred to as "neural networks" has little to do with the "neural networks" that are used in the machine learning commu…

Just curious then, how are people optimizing network topology?

Regularisation methods like dropout are often good enough that you can build a network with too many parameters (for the amount of data you have) and rely upon the regularisation to find the subset of that network that is actually useful. People have recently got good results from also randomly dropping weights, or even whole layers.

Re: Tinker with a Neural Network in Your Browser

#98

Post by DavidSJ »

wjnc wrote: post #87
eggy wrote: post #28

I started reading about ANNs in the 1980s, and had similar confusion to those here, since it was just for fun. I suggest reading a basic book or online information that goes over the basics [1]. I struggled through $200 text books, and jumped from one to the other as an autodidact. I am now studying TWEANNs (Topology and Weight Evolving Artificial Neural Networks), which basically are what you see here with the excep…

Any thoughts on why genetic programming is not 'in fashion'? Does it have anything to do with complexity of the calculations? I can imagine that the advanced models use many, many machines and only deliver results after a large training time. Genetic programming is not feasible then, if you cannot get a quick grasp of the potential results of a model.

If your program is a neural network with N parameters, or a program tree with N nodes, then testing against data takes O(N) time. With evolutionary computation, what you get for your trouble is a single real number -- the loss: how bad it did. With neural networks, backpropagation gives you N real numbers: the gradient of loss with respect to each parameter.

Put another way: with evolution you have to stumble around blindly in parameter space and rely on selection to keep you moving in the right direction. With the gradient descent that neural networks use, you get, essentially for free, knowledge of the (locally) best direction to move in parameter space.

The bigger the models, the more this matters. Modern neural networks have millions or even billions of parameters, and that's been crucial to their expressive power. Good luck learning a program tree with a billion nodes using evolution. It might take 4.54 billion years.

Re: Tinker with a Neural Network in Your Browser

#99

Post by beardicus »

erostrate wrote: post #94

The swiss roll problem also illustrates nicely the idea behind deep learning. Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d Then the deep learning guys realized that you don't have to engineer all these…

I think I learned more from your post and your two imgur links than from poking at the site for an hour. Thanks.

Would it make sense for them to add a gallery of good solutions for each problem, or would they all basically be your second example network (no time to play and see for myself right now)?

Re: Tinker with a Neural Network in Your Browser

#100

Post by findthewords »

moconnor wrote: post #88
danblick wrote: post #71

Has anyone been able to learn a function for the spiral (Swiss roll) data that's as good as a human-designed function would be?

For this simple example just choosing the largest possible fully-connected network with ReLU and L2 regularization to prevent overfit quickly converges to a nice spiral (test loss of 0.001 for me): http://playground.tensorflow.org/#activation=relu&regulariza...

I wouldn't call it quick... spiral in 150 iterations, with sigmoid magic: http://playground.tensorflow.org/#activation=sigmoid&regular...

I find the pulsating unsightly.

Post reply on HN