Gated Linear Networks
arxiv.org
Gated Linear Networks
1–10 of 31 posts
Re: Gated Linear Networks
#2Where could this shine? Could it beat SOTA on Nlp tasks?
Re: Gated Linear Networks
#3"We show that this architecture gives rise to universal learning capabilities in the limit, with effective model capacity increasing as a function of network size in a manner comparable with deep ReLU networks."
What exactly this statement means?
Re: Gated Linear Networks
#4That is an amazing paper, a great result and new neutral architectures are long overdue.
But I don't believe that this has any significance in practice.
GPU memory is the limiting factor for most current AI approaches. And that's where the typical convolutional architectures shine, because they effectively compress the input data, then work on the compressed representation, then decompress the results. With gated linear networks, I'm required to always work on the full input data, because it's a one step prediction. As the result, I'll run out of GPU memory before I reach a learning capacity that is comparable to conv nets.
Re: Gated Linear Networks
#5"We show that this architecture gives rise to universal learning capabilities in the limit, with effective model capacity increasing as a function of network size in a manner comparable with deep ReLU networks." What exactly this statement means?
Presumably (haven't read the paper yet) that their network provably becomes a universal function approximator in the limit of infinite size.
Reading... actually the proof seems to be in
Re: Gated Linear Networks
#6"We show that this architecture gives rise to universal learning capabilities in the limit, with effective model capacity increasing as a function of network size in a manner comparable with deep ReLU networks." What exactly this statement means?
As the network size increases, it can learn more complex functions. When the network gets bigger and bigger, it gets closer to being able to learn any arbitrary function.
Re: Gated Linear Networks
#7"We show that this architecture gives rise to universal learning capabilities in the limit, with effective model capacity increasing as a function of network size in a manner comparable with deep ReLU networks." What exactly this statement means?
They mean that if you add parameters, the learning capability of their approach grows by a similar amount as if you would add the same number of parameters to a conv+ReLu network (the standard approach).
That "universal" is a weird claim in my opinion, but they mean that with enough parameters, this architecture can learn everything.
Re: Gated Linear Networks
#8That is an amazing paper, a great result and new neutral architectures are long overdue. But I don't believe that this has any significance in practice. GPU memory is the limiting factor for most current AI approaches. And that's where the typical convolutional architectures shine, because they effectively compress the input data, then work on the compressed representation, then decompress the results. With gated lin…
What about findings w. R. T. Online learning? I find continuous learning quality of algorithms to be a topic which often seems to be more of a side-concern, although it carries a lot of relevance in applied settings.
Re: Gated Linear Networks
#9That is an amazing paper, a great result and new neutral architectures are long overdue. But I don't believe that this has any significance in practice. GPU memory is the limiting factor for most current AI approaches. And that's where the typical convolutional architectures shine, because they effectively compress the input data, then work on the compressed representation, then decompress the results. With gated lin…
What about findings w. R. T. Online learning? I find continuous learning quality of algorithms to be a topic which often seems to be more of a side-concern, although it carries a lot of relevance in applied settings.
I believe that to be a red herring. Their approach cannot learn any features that provide a lower-dimensional approximation of the input data. As the result, there is no intermediate representation which could change and thereby negatively affect previously learned classifiers.
But if I train 10 independent traditional networks, I also won't have newly learned data affect old performance. So in effect they give up the possibility to do transfer learning in exchange for avoiding the disadvantages of transfer learning. But that's a bad tradeoff.
With their approach you always train from scratch, which brings with it the need for huge training data sets.
So I can train a bird classifier on the traditional architecture with 500 labeled images and a pretrained resnet. Our I use a million bird images and this approach.
Re: Gated Linear Networks
#10"We show that this architecture gives rise to universal learning capabilities in the limit, with effective model capacity increasing as a function of network size in a manner comparable with deep ReLU networks." What exactly this statement means?
They mean that if you add parameters, the learning capability of their approach grows by a similar amount as if you would add the same number of parameters to a conv+ReLu network (the standard approach). That "universal" is a weird claim in my opinion, but they mean that with enough parameters, this architecture can learn everything.
I was able to get the second part of the statement. but I haven't seen the use of "in limit" in a statement like this.
Yes, the universal approximation is a strong claim. NN has been proven to have universal approximation theoretically.