Has anyone been able to learn a function for the spiral (Swiss roll) data that's as good as a human-designed function would be?
Tinker with a Neural Network in Your Browser
71–80 of 122 posts
Re: Tinker with a Neural Network in Your Browser
#72Has anyone been able to learn a function for the spiral (Swiss roll) data that's as good as a human-designed function would be?
Do you consider test loss around 0.04 good?
Re: Tinker with a Neural Network in Your Browser
#73I started reading about ANNs in the 1980s, and had similar confusion to those here, since it was just for fun. I suggest reading a basic book or online information that goes over the basics [1]. I struggled through $200 text books, and jumped from one to the other as an autodidact. I am now studying TWEANNs (Topology and Weight Evolving Artificial Neural Networks), which basically are what you see here with the excep…
As far as the recent deep learning boom is concerned, genetic programming is really out of favor. I don't really see it in any of the deep learning (or even machine learning, for that matter) literature/successes/research groups. "Neural networks" are a really really overloaded term. A ton of stuff referred to as "neural networks" has little to do with the "neural networks" that are used in the machine learning commu…
You're spot on about genetic programming. I am a self-taught person who plays with anything that strikes my fancy; I learn by playing. I read all three volumes of the Artificial Life series from the Santa Fe Institute at the time (now there are more), and went in many directions in the 1990s - Fuzzy Logic, Expert Systems, ANNs, and Evolutionary Computation (GA (Genetic Algorithms) and GP Genetic Programming), and AL (Artificial Life) all fascinating. I found, and still find, genetic programming attractive even if it has not found its niche in the ML community. I think the CI (Computational Intelligence) community at large will eventually develop well-fitted uses for it. I was trying to use an FPGA and Koza's modified GP code to have the FPGA re-program itself as a GP evolved a better program than I originally wrote to kickstart it. I didn't get too far. This was 1996-97 though. Pretty much on my own then, not really much of an Internet to find information, especially esoteric information, or cheap many-gated FPGAs.
Outside of ML, GP has found moderate success. One example is this paper (sorry behind paywall, so only the paper title here), that started with using expert data, tried ANNs, then ANNs and statistics, until it used a GP approach:
"A Computational Intelligence-Based Genetic Programming Approach for the Simulation of Soil Water Retention Curves"
I also use the term ANNs over just NNs to keep it to the silicon, and not wetware ;) Although, they did hook up a small ANN to a cockroach once, IIRC...
Re: Tinker with a Neural Network in Your Browser
#74Has anyone been able to learn a function for the spiral (Swiss roll) data that's as good as a human-designed function would be?
Re: Tinker with a Neural Network in Your Browser
#75I'm not well versed in neural networks but a lot of the new neural network software stacks coming out seem to be quite plug and plug. What kind of expertise would engineers need to have a few years from now when the technology is well developed and it doesn't need to be rewritten from scratch every time?
I'm not qualified to answer this, but I will anyway. To "operate" neural networks (as opposed to writing a framework for them), you need to know the building blocks. There are basic blocks like fully connected layers, convolutions, and nonlinear activations. Beyond those, there are higher level building blocks like LSTMs[1], gated recurrent units[2], highway layers[3], batch normalization[4], and residual blocks[5] t…
So you don't think some of these details will not be automated away in the near future so that it doesn't require a specialist to do operate a neural network?
Re: Tinker with a Neural Network in Your Browser
#76Earlier quoted context omitted.
It has its niche applications. The only non machine vision application that comes to mind is one[1] that takes a pile of data, and evolves a model that fits it. Generally were its actually being used they are a bit quiet on how they go about getting the results they do. While the genetic bit is easy, the secret sauce is in guiding learning/evolution that work for the particular problem domain. [1]: http://www.nutonia…
Yes, but all of the algorithmic advances in academia, and most of the advances at Google/Facebook, have been out in the open.
Yes, it is a shame people don't share their advances in science and technology for fear of losing market share usually. Sharing grows the market, and then there's more pie for everyone, and more work gets done to advance the field.
Still that point, nor how successful GP is in the ML community, measures its current or future potential. The book I am working my way through now, in LFE (Lisp Flavored Erlang vs. Erlang, or Elixir), is "The Handbook of Neuroevolution Through Erlang" by Gene Sher [1]
Gene covers a lot of ground. Somebody has done some transliteration to Elixir too; I use LFE, since staying with Lisp bridges the gap between my GP work, and what Gene has done with Erlang and ANNs and EC. For GP, you really need to be able to create new forms with macros, or it is more in line with GP. To quote and excerpt from Robert Virding, co-designer of Erlang, and creator of LFE,addressing Elixir's macros or messing with Erlang's modules vs. LFE's or Lisp's macros on HN before:
"There is syntactic support for making the function calls look less like function calls but the macros you define are basically function calls.
[1] http://www.erlang-factory.com/upload/presentations/536/ErlangConferencePresentation_2012.pdf
[2] https://news.ycombinator.com/item?id=7623991Re: Tinker with a Neural Network in Your Browser
#77Has anyone been able to learn a function for the spiral (Swiss roll) data that's as good as a human-designed function would be?
[deleted]
Re: Tinker with a Neural Network in Your Browser
#78Has anyone been able to learn a function for the spiral (Swiss roll) data that's as good as a human-designed function would be?
Re: Tinker with a Neural Network in Your Browser
#79Earlier quoted context omitted.
I'm not qualified to answer this, but I will anyway. To "operate" neural networks (as opposed to writing a framework for them), you need to know the building blocks. There are basic blocks like fully connected layers, convolutions, and nonlinear activations. Beyond those, there are higher level building blocks like LSTMs[1], gated recurrent units[2], highway layers[3], batch normalization[4], and residual blocks[5] t…
So you don't think some of these details will not be automated away in the near future so that it doesn't require a specialist to do operate a neural network?
I think most of those things will remain important:
+ Designing the network architecture is a means to instill your knowledge of the problem into the network. For example, using convolutions over images encodes some translational invariance into the network. It makes up for lack of data. I don't think data augmentation alone is enough, either: if you use a "stupid" architecture with heaps of data, the computation will become too expensive or slow.
- The systems engineering part will probably get automated. I bet there are Amazon engineers crying at their desks while working on AWS Elastic Tensorshift right now. So unless you're specifically interested in that side of things, maybe this isn't the best area to focus on.
+ There are always going to be problems, so knowing how to debug is a useful skill.
+ ML/stats fundamentals aren't going away. You need to know what you're trying to do before you can do it.
Re: Tinker with a Neural Network in Your Browser
#80this is very nice! I think that the reason swiss roll doesn't work as easily might be because of initialization. In 2 dimensions you have to be very careful with initializing the weights or biases because small networks get more easily stuck in bad local minima.
In this case you see that it is the swiss roll so you could say pick "proper initialization". But that technique would not work when you cannot see that it is a "swiss roll" or in multiple dimensions.
I'm pretty sure he wasn't talking about the swiss roll specifically. Big gains in neural net performance have been made through better initialization schemes (not dataset specific, just in general, e.g. an initialization scheme might adapt the initial weight distribution depending on the number of hidden units in the next layer), and smaller models are in general more sensitive to initialization.