11–20 of 23 posts
"Reservoir computing" taken very literally: wave interference in a bucket of water computed a simple speech recognition task (differentiate "zero" and "one") [1]. [1] https://link.springer.com/chapter/10.1007%2F978-3-540-39432-...
"Reservoir computing" taken very literally: wave interference in a bucket of water computed a simple speech recognition task (differentiate "zero" and "one") [1]. [1] https://link.springer.com/chapter/10.1007%2F978-3-540-39432-...
Also the Liquid State Machine paper they cited, "Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations": http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.5.8...
Fascinating idea. If I understand correctly, it's like an extreme version of data pre-processing. If you're trying to figure out whether an audio clip is saying "zero" or "one", analysing the raw amplitude data is pretty tough going. Instead you could run it through a Fourier transform in the hope that the clip's frequency content would be easier to analyse. If that doesn't help, maybe a wavelet transform, or something more fun like, uh, the "inverse Fourier transform of the logarithm of the squared magnitude of the Fourier transform"
In a sense, it doesn't really matter what pre-processing you do, provided that the differences between "zero" and "one" are more distinct in the output than the input. This is the "separation property" that the papers mention: important differences get magnified at the expense of unimportant ones. If that's true, your final analysis will have a lot less work to do.
What's cool about this is that "anything that magnifies important differences" is a pretty open-ended requirement, leaving you free to choose pre-processing that's easy to implement in hardware. In this case, fluid dynamics has the desired properties, and the laws of our universe make it very easy to implement a fluid simulation using actual fluid in an actual bucket.
Perhaps there are other systems with similar separation properties that are even easier to implement in hardware. Maybe something with electromagnetic waves, like in time-domain reflectometry? Even if such a system's behaviour is uninterpretable to us, it might still provide useful pre-processing to a machine learning algorithm.
They're expensive to train, in computing time and data-requirements, but they're the closest thing we have to trained programs.
eli5?
I am simplifying it to a level where it becomes borderline useless/wrong, but it should give a gist of the idea: it turns out that frequently you do not need to train anything but the last layer of a deep neural network, if all the other layers are sufficiently big arbitrary and weird. In reservoir computing you replace all but the last layer of the network with a "dynamical system", i.e. a large and meaningless map.
[1] http://www.physnews.com/nano-physics-news/cluster1837307157/
You should be aware that echo-state networks are considered special cases of recurrent networks. One variant of recurrent networks, LTSMs, dominates state of the art on a wide variety of problems. They're expensive to train, in computing time and data-requirements, but they're the closest thing we have to trained programs.
There are a few interesting resources about this[1]. The concept of "microelectromechanical neural network application" [2] is super interesting. We've been hearing about memristors for a while, but has anyone seen them being widely deployed anywhere? [1] http://www.physnews.com/nano-physics-news/cluster1837307157/ [2] https://aip.scitation.org/doi/full/10.1063/1.5038038
What I don't understand is if you can train a final layer on the reservoir's random representation, why is this better than just training the final layer on your data directly? I assume the answer has something to do with dimensionality reduction?
Sort of acting like a hash function - small differences in the source become large differences in the output.