Transformers Are Graph Neural Networks • Hacker News Board

Re: Transformers Are Graph Neural Networks

#11

Post by woko » Sat, Sep 12, 2020, 8:54 PM UTC

cjauvin wrote: post #4

iflp wrote: post #3

I appreciate the effort the authors put to this post, but this is like saying DNNs are stacked logistics regression: the connection is superficial, and doesn't lead to deep insights about how they really work.

I'm genuinely a bit surprised by that, that was always my high-level understanding of what the essence of neural networks was (at least feedforward vanilla ones), would you care to elaborate?

I think this post means the same as this tweet:

> Transformers are a special case of Graph Neural Networks. This may be obvious to some.

https://twitter.com/OriolVinyalsML/status/123378359362695168...

Re: Transformers Are Graph Neural Networks

#12

Post by czzr » Sat, Sep 12, 2020, 9:14 PM UTC

fizixer wrote: post #8

iflp wrote: post #3

I appreciate the effort the authors put to this post, but this is like saying DNNs are stacked logistics regression: the connection is superficial, and doesn't lead to deep insights about how they really work.

ML publication is a complete mess right now. Anyone can claim anything as long as they do a write-up and include some equations and pretty plots. It was hard enough 5 years ago to filter out handful of good papers from the sea of bad research. Now it's getting near impossible.

Somebody should train a model to do it.

Re: Transformers Are Graph Neural Networks

#13

Post by jdc » Sat, Sep 12, 2020, 10:37 PM UTC

Better explained by Chaitanya Joshi (while at the NTU Graph Deep Learning Lab)

here: https://threadreaderapp.com/thread/1233220586358181888.html

and here: https://graphdeeplearning.github.io/post/transformers-are-gn...

Re: Transformers Are Graph Neural Networks

#14

Post by nmca » Sat, Sep 12, 2020, 11:33 PM UTC

fpgaminer wrote: post #9

It's interesting watching these attempts to understand Transformers. Are they Graph networks? Are they Hopfields networks? Are they convolutions? There was this research: https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreti... Turns out, GPT-like architectures appear to use the same representation throughout all the layers. So you can use the final head layer as a lens to see what words the network is thinkin…

You might be interested in RTRL (real time recurrent learning).

Re: Transformers Are Graph Neural Networks

#15

Post by YetAnotherNick » Sat, Sep 12, 2020, 11:42 PM UTC

fpgaminer wrote: post #9

It's interesting watching these attempts to understand Transformers. Are they Graph networks? Are they Hopfields networks? Are they convolutions? There was this research: https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreti... Turns out, GPT-like architectures appear to use the same representation throughout all the layers. So you can use the final head layer as a lens to see what words the network is thinkin…

Actually we do have few mechanisms for long term memory like Neural Turing Machine, which has explicit memory cells which neural network could read and write. I think the only thing that is holding back NTM is that it is computationally not efficient like fixed sized context transformer.

Re: Transformers Are Graph Neural Networks

#16

Post by sjg007 » Sun, Sep 13, 2020, 1:07 AM UTC

fpgaminer wrote: post #9

It's interesting watching these attempts to understand Transformers. Are they Graph networks? Are they Hopfields networks? Are they convolutions? There was this research: https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreti... Turns out, GPT-like architectures appear to use the same representation throughout all the layers. So you can use the final head layer as a lens to see what words the network is thinkin…

When I realized that what transformers do is transform input into output which is also input I was amazed but it makes sense. It’s exactly like a markov chain. Think of a snake eating itself. What’s important is that the output is basically a Probability distribution. Now you can post process that output to get a finite value but you really want to put it back in and turn the wheel again.

But you are right, they are trained on next word prediction so there’s no long term memory. I imagine people are working on transformers with a memory bank. But RNNs seem to be the brute force solution here... what I am guessing is that you need to maintain some kind of index to decide where to backprop. If it hasn’t been discovered yet, I bet it will be some kind of bloom filter.

Re: Transformers Are Graph Neural Networks

#17

Post by iflp » Sun, Sep 13, 2020, 5:08 AM UTC

cjauvin wrote: post #4

iflp wrote: post #3

I appreciate the effort the authors put to this post, but this is like saying DNNs are stacked logistics regression: the connection is superficial, and doesn't lead to deep insights about how they really work.

I'm genuinely a bit surprised by that, that was always my high-level understanding of what the essence of neural networks was (at least feedforward vanilla ones), would you care to elaborate?

It depends on what kind of understanding you want to achieve. It can be helpful to think of DNN as approximating the corresponding infinitely-wide versions. Depending on how you deal with certain scaling, they then act like a linear filter of the error signal in function space, or for single-hidden-layer networks at least, an interacting particle system. In both cases you can understand the convergence of gradient descent training using these analogies, although gaps from real-world practice exist.

Re: Transformers Are Graph Neural Networks

#18

Post by Erlich_Bachman » Sun, Sep 13, 2020, 7:19 AM UTC

fpgaminer wrote: post #9

It's interesting watching these attempts to understand Transformers. Are they Graph networks? Are they Hopfields networks? Are they convolutions? There was this research: https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreti... Turns out, GPT-like architectures appear to use the same representation throughout all the layers. So you can use the final head layer as a lens to see what words the network is thinkin…

Very well thought-out! We need more of this. Doesn't mean that all of your ideas are correct or don't make unnecessary assumptions, etc. But it shows a very clear thinking path that is easy to understand. You should write and publish more if you can!

Re: Transformers Are Graph Neural Networks

#19

Post by postfacto » Sun, Sep 13, 2020, 7:32 AM UTC

I thought Transformers were “Robots in disguise”.

Re: Transformers Are Graph Neural Networks

#20

Post by Muller20 » Sun, Sep 13, 2020, 7:50 AM UTC

YetAnotherNick wrote: post #15

fpgaminer wrote: post #9

It's interesting watching these attempts to understand Transformers. Are they Graph networks? Are they Hopfields networks? Are they convolutions? There was this research: https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreti... Turns out, GPT-like architectures appear to use the same representation throughout all the layers. So you can use the final head layer as a lens to see what words the network is thinkin…

Actually we do have few mechanisms for long term memory like Neural Turing Machine, which has explicit memory cells which neural network could read and write. I think the only thing that is holding back NTM is that it is computationally not efficient like fixed sized context transformer.

What's holding back NTM is that they are hard to train, even worse than RNNs. They are not much less efficient than a Transformer. Instead, Transformer has all the advantages of the NTM but it is much easier to train.

Actually, the way I see it, Transformer is a direct descendent of memory-based architectures (NTM, MemNet, stack-based RNNs...) that is both expressive and easy to train.