Reservoir computing
en.wikipedia.org
Reservoir computing
1–10 of 23 posts
Re: Reservoir computing
#2eli5?
Re: Reservoir computing
#3eli5?
I am simplifying it to a level where it becomes borderline useless/wrong, but it should give a gist of the idea: it turns out that frequently you do not need to train anything but the last layer of a deep neural network, if all the other layers are sufficiently big arbitrary and weird. In reservoir computing you replace all but the last layer of the network with a "dynamical system", i.e. a large and meaningless map.
Re: Reservoir computing
#4eli5?
https://www.nature.com/articles/ncomms1476?WT.ec_id=NCOMMS-2...
Seems like a decent introduction (not versed in the topic, but this is a more informative article than the wikipedia page).
Check out Figure 1 for a model like what krastanov discusses.
Re: Reservoir computing
#5eli5?
I am simplifying it to a level where it becomes borderline useless/wrong, but it should give a gist of the idea: it turns out that frequently you do not need to train anything but the last layer of a deep neural network, if all the other layers are sufficiently big arbitrary and weird. In reservoir computing you replace all but the last layer of the network with a "dynamical system", i.e. a large and meaningless map.
[deleted]
Re: Reservoir computing
#6"Reservoir computing" taken very literally: wave interference in a bucket of water computed a simple speech recognition task (differentiate "zero" and "one") [1].
[1] https://link.springer.com/chapter/10.1007%2F978-3-540-39432-...
Re: Reservoir computing
#7eli5?
I am simplifying it to a level where it becomes borderline useless/wrong, but it should give a gist of the idea: it turns out that frequently you do not need to train anything but the last layer of a deep neural network, if all the other layers are sufficiently big arbitrary and weird. In reservoir computing you replace all but the last layer of the network with a "dynamical system", i.e. a large and meaningless map.
Thanks for the eli5! Helpful
Re: Reservoir computing
#8eli5?
I am simplifying it to a level where it becomes borderline useless/wrong, but it should give a gist of the idea: it turns out that frequently you do not need to train anything but the last layer of a deep neural network, if all the other layers are sufficiently big arbitrary and weird. In reservoir computing you replace all but the last layer of the network with a "dynamical system", i.e. a large and meaningless map.
So treat it like a basis expansion?
Re: Reservoir computing
#9Earlier quoted context omitted.
I am simplifying it to a level where it becomes borderline useless/wrong, but it should give a gist of the idea: it turns out that frequently you do not need to train anything but the last layer of a deep neural network, if all the other layers are sufficiently big arbitrary and weird. In reservoir computing you replace all but the last layer of the network with a "dynamical system", i.e. a large and meaningless map.
So treat it like a basis expansion?
If I understand the question correctly, ie is a reservoir computing approach simply projecting an input vector into a higher dimensional space, like a wonky support vector machine, then I think the answer is: unclear.
Reservoir computing approaches usually have the intermediate layers be recurrent, eg they implement a dynamical system. Theoretically, this is actually Turing complete, although good luck programming it. But in any case, the range of behaviors — transformations in the data — that a dynamical system can implement is much more powerful than just a basis expansion. However, whether this is what is happening in actual practice is really, really unclear to me. A lot of recurrent neural networks aren't doing anything more than an equivalent feedforward network, and the same thing may be true here: reservoir approaches might, for the most part, really just be performing nonlinear projection to higher dimensional spaces, and then the output layer is being trained to classify those patterns.
Re: Reservoir computing
#10Earlier quoted context omitted.
So treat it like a basis expansion?
If I understand the question correctly, ie is a reservoir computing approach simply projecting an input vector into a higher dimensional space, like a wonky support vector machine, then I think the answer is: unclear. Reservoir computing approaches usually have the intermediate layers be recurrent, eg they implement a dynamical system. Theoretically, this is actually Turing complete, although good luck programming it…
Thanks, that makes sense.