Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stacked Approximated Regression Machine: A Simple Deep Learning Approach

arxiv.org

Post reply on HN
Print view View original

11–20 of 28 posts

Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach

#11

Post by joe_the_user »

cs702 wrote: post #2

As far as I understand this, these guys claim they can train convolutional and many other types of deep neural nets faster by pretraining each layer with a new unsupervised technique via which the layer sort of learns to compress its inputs (a local optimization problem), and then they fine tune the whole network end-to-end with supervised SGD and backpropagation as usual. They have not released code, so no one else…

The paper itself is fairly sparse but references a number of approaches for more-quickly-learning neural-net-related learning systems from ~2013 (PCAnet, SCATnet,etc).

The paper presents these and other approaches as being instances of a classical, general form, regularized regression but with the "stacked" property involving each layer iterating however many times and then the next layer changing parameters (or features) and further iterating.

From my barely-informed viewpoint, this sounds like a fascinating way to unify the earlier efforts and one which could yield a variety of other approaches - even if the particular variation they use doesn't work out. But I assume lots of more-informed people are going to be looking at this.

Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach

#12

Post by jcannell »

cs702 wrote: post #2

As far as I understand this, these guys claim they can train convolutional and many other types of deep neural nets faster by pretraining each layer with a new unsupervised technique via which the layer sort of learns to compress its inputs (a local optimization problem), and then they fine tune the whole network end-to-end with supervised SGD and backpropagation as usual. They have not released code, so no one else…

This is a pretty good summary, but to note this is not a new unsupervised technique - it's a sparse coding method, and sparse coding has been around for more than a decade. Summarizing this paper without mentioning the word sparse coding seems wrong somehow. ;)

Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach

#13

Post by cs702 »

jcannell wrote: post #12
cs702 wrote: post #2

As far as I understand this, these guys claim they can train convolutional and many other types of deep neural nets faster by pretraining each layer with a new unsupervised technique via which the layer sort of learns to compress its inputs (a local optimization problem), and then they fine tune the whole network end-to-end with supervised SGD and backpropagation as usual. They have not released code, so no one else…

This is a pretty good summary, but to note this is not a new unsupervised technique - it's a sparse coding method, and sparse coding has been around for more than a decade. Summarizing this paper without mentioning the word sparse coding seems wrong somehow. ;)

You're right, sparse coding is not new -- and neither is the idea of greedy layer-wise training.

But if this works as advertised, these guys must be doing something new.

Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach

#14

Post by gabrielgoh »

billconan wrote: post #7

is there a book you can recommend about the fundamentals (like sparse coding) to understand papers like this?

The fundamentals are pretty easy. I wrote a blog post on this, let me know if it helps

http://gabgoh.github.io/SARG/

Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach

#15

Post by billconan »

gabrielgoh wrote: post #14
billconan wrote: post #7

is there a book you can recommend about the fundamentals (like sparse coding) to understand papers like this?

The fundamentals are pretty easy. I wrote a blog post on this, let me know if it helps http://gabgoh.github.io/SARG/

Thank you, I will read it! (hey! I'm also from UCDavis!)

Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach

#16

Post by gabrielgoh »

billconan wrote: post #15
gabrielgoh wrote: post #14

Earlier quoted context omitted.

The fundamentals are pretty easy. I wrote a blog post on this, let me know if it helps http://gabgoh.github.io/SARG/

Thank you, I will read it! (hey! I'm also from UCDavis!)

woop, a fellow aggie! feel free to email me if you have any questions, i want to make the post as clear as possible

Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach

#18

Post by joe_the_user »

cs702 wrote: post #10
cs702 wrote: post #4

Earlier quoted context omitted.

I'm less skeptical than fchollet (creator of Keras, for those here who don't know), but agree that we need to wait until the usual suspects at Google, FaceBook, Toronto, Montreal, Stanford, etc. have replicated this. In all likelihood the team will release code soon, either before or after NIPS, so we will all be able to check things out for ourselves.

One of the authors, Zhangyang Wang, just wrote this on his personal page: "We have discussed and decided to work on a software package release, perhaps accompanying it with a more detailed technical report in the future. Once the software package is ready, we will update everybody." http://www.atlaswang.com/

Paper withdrawn

https://arxiv.org/abs/1608.04062

It's kind of an odd thing. I (random non-academic amateur) actually spent a bunch of time trying to parse the paper, which was kind of a combination of interesting ideas and incomprehensible ambiguities.

One real academic researcher also put some time into it. The good part of the paper is explained here. My guess is the problem is going from ARM to SARM.

http://gabgoh.github.io/SARG/

While I'm sure most people involved think of the experience as a wash, I feel like I learned a bunch about deep learning in the process.

PS, also sad that the author did this.

Re: Stacked Approximated Regression Machine: A Simple Deep Learning Approach

#19

Post by Houshalter »

cs702 wrote: post #2

As far as I understand this, these guys claim they can train convolutional and many other types of deep neural nets faster by pretraining each layer with a new unsupervised technique via which the layer sort of learns to compress its inputs (a local optimization problem), and then they fine tune the whole network end-to-end with supervised SGD and backpropagation as usual. They have not released code, so no one else…

>For example, we now know that deep neural nets can be trained with approximate gradients obtained by shifting bits to get the sign and order of magnitude of the gradient roughly right.[1] In some cases it's even possible to restrict learning to use binary weights.[2] More recently, we have learned that it's possible to use "helper" linear models during training to predict what the gradients will be for each layer, in-between true-gradient updates, allowing layers to update their parameters locally during backpropagation.[3] Finally, don't forget that in the late 2000's, AI researchers were doing a lot of interesting work with unsupervised layer-wise training (e.g., DBNs composed of RBMs, stacked autoencoders).[4]

All of those use fundamentally use back propagation though, they just approximate the gradients. Pretraining with autoencoders even used back propagation to train the autoencoders. And researchers have entirely moved away from that strategy because it just doesn't work as well as pure supervised learning.

I don't think backpropagation will ever go away because it is so simple and powerful. Instead it will be tweaked and approximated like in your examples.

Post reply on HN