Tinker with a Neural Network in Your Browser • Hacker News Board

Re: Tinker with a Neural Network in Your Browser

#111

Post by chestervonwinch » Wed, Apr 13, 2016, 1:44 PM UTC

espadrine wrote: post #108

chestervonwinch wrote: post #105

Earlier quoted context omitted.

Using their network, you are limited to 8 units per layer it seems. So, I ported their swiss roll dataset to python and threw together a shallow network trainer with theano: https://gist.github.com/notmatthancock/68d52af2e8cde7fbff1c9... Then, I trained a shallow network with 36 hidden units (your deep net has 6 units and 6 layers): http://i.imgur.com/I0pXaTK.png edit: I forgot to mention that the shallow network abo…

Just so I understand correctly: your network has 100000 iterations, while the parent's has 1000, but they both only use x / y positions? It feels like neurons in the first layer are weaker, because all they can do is a linear separation. Given deep networks, I was wondering if adding neurons to the first layer was better than adding them to the last one, and empirically, it feels like it is quite worse. I wonder if t…

> your network has 100000 iterations, while the parent's has 1000, but they both only use x / y positions

Correct, but keep in mind that their method appears to use batch descent while mine does not. Batch descent is often converges more quickly. There are other differences between my net and the GP's I can spot as well (e.g., the activation function, the learning rate, and regularization).

Also keep in mind that I threw this together over breakfast, and did not spend much time tweaking parameters :)

Re: Tinker with a Neural Network in Your Browser

#112

Post by romaniv » Wed, Apr 13, 2016, 3:13 PM UTC

erostrate wrote: post #94

The swiss roll problem also illustrates nicely the idea behind deep learning. Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d Then the deep learning guys realized that you don't have to engineer all these…

>Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc.

It's probably worth pointing out that this is true for ANNs, but there were (and are) other "shallow" classifiers that can handle swiss roll problem without manual parameter encoding. SVMs, for example.

http://cs.stanford.edu/people/karpathy/svmjs/demo/

Re: Tinker with a Neural Network in Your Browser

#113

Post by rolandog » Wed, Apr 13, 2016, 3:37 PM UTC

erostrate wrote: post #94

The swiss roll problem also illustrates nicely the idea behind deep learning. Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d Then the deep learning guys realized that you don't have to engineer all these…

What I found interesting is that I couldn't get a proper fit with the same parameters you showed... however, I could 'speed up' the learning by regenerating the data during the learning process.

It may just be that 'batched cumulative learning' (I don't know if there is already a term for this) gets a better fit than just learning from a smaller set of data.

Edit: Did a quick test; regenerating about every 50 and 100 iterations, and conversion does seem faster (at least, when a clear spiral is formed). https://imgur.com/a/OPjXb

Re: Tinker with a Neural Network in Your Browser

#114

Post by haddr » Wed, Apr 13, 2016, 5:14 PM UTC

ylem wrote: post #50

argonaut wrote: post #40

Earlier quoted context omitted.

As far as the recent deep learning boom is concerned, genetic programming is really out of favor. I don't really see it in any of the deep learning (or even machine learning, for that matter) literature/successes/research groups. "Neural networks" are a really really overloaded term. A ton of stuff referred to as "neural networks" has little to do with the "neural networks" that are used in the machine learning commu…

Just curious then, how are people optimizing network topology?

Probably also through some grid search. I've read (but not rememeber where) that Random Search gives very good results, even better than grid (in less time).

Re: Tinker with a Neural Network in Your Browser

#115

Post by Your_Creator » Wed, Apr 13, 2016, 5:17 PM UTC

So glad anns are becoming mainstream

Eventually it will have to be recognized as a new species of life, so I hope programmers, tinkerers and everyone else keeps that in mind because all life must be respected

And this particular form will be our responsibility, we can either embrace it as we continue to merge with our technology, or we can allow ourselves to go extinct like so many other species already have

For the naysayers - ever notice how attached we are to our phones? Many behave as if they are missing a limb without it - it's because they are, the brain adapts rapidly and for many, the brain has adapted to outsourcing our cognition. It used to be books, day runners, journals, diaries - now we have devices and soon they'll be implants or prosthetics

The writers at marvel who came up with the idea of calling iron man's suit a prosthetic were definately onto something and suits like that are probably our best chance of successful colonization of other planets. We'll need ai to be our friend out there, working with us

Re: Tinker with a Neural Network in Your Browser

#116

Post by espadrine » Wed, Apr 13, 2016, 6:29 PM UTC

rolandog wrote: post #113

erostrate wrote: post #94

The swiss roll problem also illustrates nicely the idea behind deep learning. Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d Then the deep learning guys realized that you don't have to engineer all these…

What I found interesting is that I couldn't get a proper fit with the same parameters you showed... however, I could 'speed up' the learning by regenerating the data during the learning process. It may just be that 'batched cumulative learning' (I don't know if there is already a term for this) gets a better fit than just learning from a smaller set of data. Edit: Did a quick test; regenerating about every 50 and 100…

Regenerating the data is kind of cheating; it is as if you were given twice the amount of data.

In a normal situation, you obtain a list of input / output (say, images as input, a digit as output, for learning handwritten digits). You separate it between training data (which actually improves the net) and testing data (to detect overfitting), and you don't get more data than that.

Here, you can generate more data for free, as we have the function we want to approximate. Having more data will often result in a better result and faster convergence.

Re: Tinker with a Neural Network in Your Browser

#117

Post by asab » Wed, Apr 13, 2016, 6:33 PM UTC

asab wrote: post #78

danblick wrote: post #71

Has anyone been able to learn a function for the spiral (Swiss roll) data that's as good as a human-designed function would be?

http://playground.tensorflow.org/#activation=tanh&batchSize=...

Update: after playing with this for way too long, I've found that it can converge to a spiral with 3 or 2 or even just 1 node in the 2nd hidden layer.

The 1 node case is especially interesting, because when it converges the single node must learn the whole spiral pattern. Although with noise it can be less reliable with more jagged edges, as well as take longer to converge (also bumped the learning rate down), seeing the spiral encoded directly in the 2nd hidden layer is more interesting to me.

http://playground.tensorflow.org/#activation=tanh&batchSize=...

Re: Tinker with a Neural Network in Your Browser

#118

Post by eggy » Thu, Apr 14, 2016, 12:17 AM UTC

wjnc wrote: post #87

eggy wrote: post #28

I started reading about ANNs in the 1980s, and had similar confusion to those here, since it was just for fun. I suggest reading a basic book or online information that goes over the basics [1]. I struggled through $200 text books, and jumped from one to the other as an autodidact. I am now studying TWEANNs (Topology and Weight Evolving Artificial Neural Networks), which basically are what you see here with the excep…

Any thoughts on why genetic programming is not 'in fashion'? Does it have anything to do with complexity of the calculations? I can imagine that the advanced models use many, many machines and only deliver results after a large training time. Genetic programming is not feasible then, if you cannot get a quick grasp of the potential results of a model.

First, the right tool for the job. ANNs are able to be a general function approximator with sufficient training to be a cost-effective choice to implement. Second, ANNs have been around about 35 years longer than GP. The TWEANNs I am studying, and that I already mentioned in a previous reply in this thread, hybridize ANNs and EC (GAs and GP), so if you include Neural Networks that utilize Evolutionary Computation techniques to modify weights or topology, then GP is being used to an extent. Replication as a variable in EC is the key force in biology, and I only see more use of EC techniques to enhance the general function approximators that are ANNs. Further, there are also hybridized computing machines that have been made, and are being made with FPGAs and GPUs. Finance and supercomputing are just two areas that are looking to utilize them. In some, the FPGAs are simply there for updating special computation programs that feed the GPUs. There is some research with a GP optimizer updating the FPGAs and then using the GPUs for the massive parallelization of the computations.

Re: Tinker with a Neural Network in Your Browser

#119

Post by wjnc » Thu, Apr 14, 2016, 5:27 AM UTC

eggy wrote: post #118

wjnc wrote: post #87

Earlier quoted context omitted.

Any thoughts on why genetic programming is not 'in fashion'? Does it have anything to do with complexity of the calculations? I can imagine that the advanced models use many, many machines and only deliver results after a large training time. Genetic programming is not feasible then, if you cannot get a quick grasp of the potential results of a model.

First, the right tool for the job. ANNs are able to be a general function approximator with sufficient training to be a cost-effective choice to implement. Second, ANNs have been around about 35 years longer than GP. The TWEANNs I am studying, and that I already mentioned in a previous reply in this thread, hybridize ANNs and EC (GAs and GP), so if you include Neural Networks that utilize Evolutionary Computation tec…

Thanks eggy, awesome replys. You should write some of your experiences down if you find the time.

Re: Tinker with a Neural Network in Your Browser

#120

Post by therein » Thu, Apr 14, 2016, 7:20 AM UTC

amelius wrote: post #102

erostrate wrote: post #94

The swiss roll problem also illustrates nicely the idea behind deep learning. Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d Then the deep learning guys realized that you don't have to engineer all these…

But how will the number of neurons N grow with the number of turns in the spiral? If N levels off, then the network has grasped the concept of a spiral and can generalize to arbitrary size. If N doesn't level off, then the network isn't really learning the general case.

I know this is going to sound cheesy but that's an amazing way to put it. It blew my mind.