Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gated Linear Networks

arxiv.org

Post reply on HN
Print view View original

11–20 of 31 posts

Re: Gated Linear Networks

#11

Post by fxtentacle »

Immortal333 wrote: post #10
fxtentacle wrote: post #7

Earlier quoted context omitted.

They mean that if you add parameters, the learning capability of their approach grows by a similar amount as if you would add the same number of parameters to a conv+ReLu network (the standard approach). That "universal" is a weird claim in my opinion, but they mean that with enough parameters, this architecture can learn everything.

I was able to get the second part of the statement. but I haven't seen the use of "in limit" in a statement like this. Yes, the universal approximation is a strong claim. NN has been proven to have universal approximation theoretically.

"in limit" here means as you approach the edge case of having an unlimited number of parameters.

Re: Gated Linear Networks

#12

Post by janosett »

Immortal333 wrote: post #3

"We show that this architecture gives rise to universal learning capabilities in the limit, with effective model capacity increasing as a function of network size in a manner comparable with deep ReLU networks." What exactly this statement means?

> "We show that this architecture"

They are demonstrating a new technique, Gated Linear Networks.

> gives rise to universal learning capabilities in the limit

they claim to show that with an unbounded amount of time and memory (network size / # params) this architecture can be used to learn/approximate any function

> with effective model capacity increasing as a function of network size

Model capacity here refers to the ability to memorize a mapping between inputs and outputs. They show that a network with more layers/weights will "memorize" more.

> in a manner comparable with deep ReLU networks

"Deep ReLU networks" are referring to commonly used modern deep neural network architectures. ReLU is a popular activation function: https://en.wikipedia.org/wiki/Rectifier_(neural_networks)

Re: Gated Linear Networks

#13

Post by friendly_aixi »

fxtentacle wrote: post #4

That is an amazing paper, a great result and new neutral architectures are long overdue. But I don't believe that this has any significance in practice. GPU memory is the limiting factor for most current AI approaches. And that's where the typical convolutional architectures shine, because they effectively compress the input data, then work on the compressed representation, then decompress the results. With gated lin…

Convolution is a linear operation; in the case of images, you can view it as a multiplication with a doubly block circulant matrix. I can't see any barriers to hybrid approaches here, though it seems difficult to avoid using backpropagation for credit assignment within the convolutional layers.

Re: significance, how about their application in regression: https://arxiv.org/abs/2006.05964 ? Or in contextual bandits: https://arxiv.org/abs/2002.11611 ?

Disclaimer: I am one of the authors (Joel).

Re: Gated Linear Networks

#14

Post by friendly_aixi »

Immortal333 wrote: post #10
fxtentacle wrote: post #7

Earlier quoted context omitted.

They mean that if you add parameters, the learning capability of their approach grows by a similar amount as if you would add the same number of parameters to a conv+ReLu network (the standard approach). That "universal" is a weird claim in my opinion, but they mean that with enough parameters, this architecture can learn everything.

I was able to get the second part of the statement. but I haven't seen the use of "in limit" in a statement like this. Yes, the universal approximation is a strong claim. NN has been proven to have universal approximation theoretically.

The result here is stronger, in the sense that typical NN universality results are statements with respect to just capacity (and not how you optimise them). Here, the result holds with respect to both capacity + a choice of suitable no regret online convex optimisation algorithm (e.g. online gradient descent). Of course, this is just one desirable property of a general purpose learning algorithm.

Re: Gated Linear Networks

#15

Post by friendly_aixi »

fxtentacle wrote: post #9
BrokrnAlgorithm wrote: post #8

Earlier quoted context omitted.

What about findings w. R. T. Online learning? I find continuous learning quality of algorithms to be a topic which often seems to be more of a side-concern, although it carries a lot of relevance in applied settings.

I believe that to be a red herring. Their approach cannot learn any features that provide a lower-dimensional approximation of the input data. As the result, there is no intermediate representation which could change and thereby negatively affect previously learned classifiers. But if I train 10 independent traditional networks, I also won't have newly learned data affect old performance. So in effect they give up th…

1) Indeed, GLNs don't learn features... but I would claim they do learn some notion of an intermediate representation, it's just different from the DL mainstream -- in particular its closely related to the inverse Radon transform in medical imaging.

2) Inputs which are similar in terms of cosine similarity will map to similar (data dependent) products of weight matrices, and thus behave similarly, which of course can affect performance in both good and bad ways. With the results we show on permuted MNIST, its well... just not particularly likely that they will interfere. This is a good thing -- why should completely different data distributions interfere with one another? The point is the method is resiliant to catastrophic forgetting when the cosine similarity between data items from different tasks is small. This highlights the different kind of inductive bias a halfspace gated GLN has compared to a Deep ReLu network.

3) Re bird example, that's slightly unfair. I am sure one could easily make use of the pre-trained resnet to provide informative features to a GLN -- it's early days for this method, hybrid systems haven't been investigated, so I don't know whether it would work better than current SOTA methods for image classification. But I would be pretty confident that some simple combination would work better than chopping the head off a pretrained network and fitting an SVM on top. This is all speculation on my part though. :)

Re: Gated Linear Networks

#16

Post by BrokrnAlgorithm »

fxtentacle wrote: post #9
BrokrnAlgorithm wrote: post #8

Earlier quoted context omitted.

What about findings w. R. T. Online learning? I find continuous learning quality of algorithms to be a topic which often seems to be more of a side-concern, although it carries a lot of relevance in applied settings.

I believe that to be a red herring. Their approach cannot learn any features that provide a lower-dimensional approximation of the input data. As the result, there is no intermediate representation which could change and thereby negatively affect previously learned classifiers. But if I train 10 independent traditional networks, I also won't have newly learned data affect old performance. So in effect they give up th…

Still, there are a lot of domains where transfer learning is no the most applicable setting - I'm thinking of highly noisy and non-stationary setting such as finance. In some of these domains, especially time series, lack of data is often not the issue, e.g. high frequency datasets.

Having models constantly re-train as the default setting is essentially what a rolling regression would do - having a rolling regression that doesn't catastrophically forget would be quite valuable.

Re: Gated Linear Networks

#17

Post by BrokrnAlgorithm »

friendly_aixi wrote: post #15
fxtentacle wrote: post #9

Earlier quoted context omitted.

I believe that to be a red herring. Their approach cannot learn any features that provide a lower-dimensional approximation of the input data. As the result, there is no intermediate representation which could change and thereby negatively affect previously learned classifiers. But if I train 10 independent traditional networks, I also won't have newly learned data affect old performance. So in effect they give up th…

1) Indeed, GLNs don't learn features... but I would claim they do learn some notion of an intermediate representation, it's just different from the DL mainstream -- in particular its closely related to the inverse Radon transform in medical imaging. 2) Inputs which are similar in terms of cosine similarity will map to similar (data dependent) products of weight matrices, and thus behave similarly, which of course can…

Good point as well - sometimes its not about dimensionality reduction but more about persistent representation, having this geared towards highly non-stationary environments is nice thing to have.

Re: Gated Linear Networks

#18

Post by jacksnipe »

Immortal333 wrote: post #10
fxtentacle wrote: post #7

Earlier quoted context omitted.

They mean that if you add parameters, the learning capability of their approach grows by a similar amount as if you would add the same number of parameters to a conv+ReLu network (the standard approach). That "universal" is a weird claim in my opinion, but they mean that with enough parameters, this architecture can learn everything.

I was able to get the second part of the statement. but I haven't seen the use of "in limit" in a statement like this. Yes, the universal approximation is a strong claim. NN has been proven to have universal approximation theoretically.

"In limit" is just shorthand for "as N tends to infinity" (not necessarily N, but you get the idea).

Re: Gated Linear Networks

#20

Post by 0-_-0 »

friendly_aixi wrote: post #13
fxtentacle wrote: post #4

That is an amazing paper, a great result and new neutral architectures are long overdue. But I don't believe that this has any significance in practice. GPU memory is the limiting factor for most current AI approaches. And that's where the typical convolutional architectures shine, because they effectively compress the input data, then work on the compressed representation, then decompress the results. With gated lin…

Convolution is a linear operation; in the case of images, you can view it as a multiplication with a doubly block circulant matrix. I can't see any barriers to hybrid approaches here, though it seems difficult to avoid using backpropagation for credit assignment within the convolutional layers. Re: significance, how about their application in regression: https://arxiv.org/abs/2006.05964 ? Or in contextual bandits: ht…

I like the choice of user name
Post reply on HN