Stacked Approximated Regression Machine: A Simple Deep Learning Approach
1–10 of 28 posts
Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach
#2As far as I understand this, these guys claim they can train convolutional and many other types of deep neural nets faster by pretraining each layer with a new unsupervised technique via which the layer sort of learns to compress its inputs (a local optimization problem), and then they fine tune the whole network end-to-end with supervised SGD and backpropagation as usual. They have not released code, so no one else has replicated this yet -- as far as I know.
If the claim holds, the implication is that layers can quickly learn much of what they need to learn locally, that is, without requiring backpropagation of gradients from potentially very distant layers. I can't help but wonder if this opens the door for more efficient asynchronous/parallel/distributed training of layers, potentially leading to models that update themselves continuously (i.e., "online" instead of in a batch process).
I wouldn't be surprised if the claim holds. There is mounting evidence that standard end-to-end backpropagation is a rather inefficient learning mechanism. For example, we now know that deep neural nets can be trained with approximate gradients obtained by shifting bits to get the sign and order of magnitude of the gradient roughly right.[1] In some cases it's even possible to restrict learning to use binary weights.[2] More recently, we have learned that it's possible to use "helper" linear models during training to predict what the gradients will be for each layer, in-between true-gradient updates, allowing layers to update their parameters locally during backpropagation.[3] Finally, don't forget that in the late 2000's, AI researchers were doing a lot of interesting work with unsupervised layer-wise training (e.g., DBNs composed of RBMs, stacked autoencoders).[4]
This is a fascinating area of research with potentially huge payoffs. For example, it would be really neat if we find there's a "general" algorithm via which layers can learn locally from inputs continuously ("online"), allowing us to combine layers into deep neural nets for specific tasks as needed.
[1] https://arxiv.org/abs/1510.03009
[2] https://arxiv.org/abs/1602.02830
[3] https://deepmind.com/blog#decoupled-neural-interfaces-using-...
[4] https://www.iro.umontreal.ca/~lisa/pointeurs/TR1312.pdf
EDITS: Expanded the original comment so it conveys better what I actually meant to write, while keeping language as casual and informal as possible. Also, I softened the tone of my more speculative observations.
Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach
#3I'm going to check back later to see if anyone manages to reproduce it. Perhaps by the time it's presented at NIPS.
A twitter conversation reflecting some scepticism, but agreeing it would be interesting if it all checks out: https://twitter.com/fchollet/status/771862837819867136
Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach
#4I'm going to check back later to see if anyone manages to reproduce it. Perhaps by the time it's presented at NIPS. A twitter conversation reflecting some scepticism, but agreeing it would be interesting if it all checks out: https://twitter.com/fchollet/status/771862837819867136
I'm less skeptical than fchollet (creator of Keras, for those here who don't know), but agree that we need to wait until the usual suspects at Google, FaceBook, Toronto, Montreal, Stanford, etc. have replicated this. In all likelihood the team will release code soon, either before or after NIPS, so we will all be able to check things out for ourselves.
Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach
#5As far as I understand this, these guys claim they can train convolutional and many other types of deep neural nets faster by pretraining each layer with a new unsupervised technique via which the layer sort of learns to compress its inputs (a local optimization problem), and then they fine tune the whole network end-to-end with supervised SGD and backpropagation as usual. They have not released code, so no one else…
Thanks for the background info.
If the layers are learned locally, can it take into account interactions between layers?
Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach
#6As far as I understand this, these guys claim they can train convolutional and many other types of deep neural nets faster by pretraining each layer with a new unsupervised technique via which the layer sort of learns to compress its inputs (a local optimization problem), and then they fine tune the whole network end-to-end with supervised SGD and backpropagation as usual. They have not released code, so no one else…
Thanks for the background info. If the layers are learned locally, can it take into account interactions between layers?
Locally != independently; the training process is recursive, so interactions between layers are present. From the paper:
"The parameters of the entire SARM are solved recursively. The current ARM’s parameters are calculated using the output from the previous ARM. Then, the output of the current ARM is fed into the subsequent ARM (or the classifier, if the current one is the last ARM), as its input."
Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach
#7is there a book you can recommend about the fundamentals (like sparse coding) to understand papers like this?
Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach
#8is there a book you can recommend about the fundamentals (like sparse coding) to understand papers like this?
Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach
#9is there a book you can recommend about the fundamentals (like sparse coding) to understand papers like this?
Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach
#10I'm going to check back later to see if anyone manages to reproduce it. Perhaps by the time it's presented at NIPS. A twitter conversation reflecting some scepticism, but agreeing it would be interesting if it all checks out: https://twitter.com/fchollet/status/771862837819867136
I'm less skeptical than fchollet (creator of Keras, for those here who don't know), but agree that we need to wait until the usual suspects at Google, FaceBook, Toronto, Montreal, Stanford, etc. have replicated this. In all likelihood the team will release code soon, either before or after NIPS, so we will all be able to check things out for ourselves.
One of the authors, Zhangyang Wang, just wrote this on his personal page: "We have discussed and decided to work on a software package release, perhaps accompanying it with a more detailed technical report in the future. Once the software package is ready, we will update everybody."