Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do vision transformers see like convolutional neural networks?

arxiv.org

Post reply on HN
Print view View original

21–30 of 45 posts

Re: Do vision transformers see like convolutional neural networks?

#21

Post by dpflan »

PartiallyTyped wrote: post #18
stared wrote: post #13

I am much more interested if they fall for the same tricks. For example, if it is easy to fool them with optical illusions, such as innocent images that look racy at the first glance: https://medium.com/@marekkcichy/does-ai-have-a-dirty-mind-to... CW: Even though it does not contain a single explicit picture, it might be considered NSFW (literally - as at the first glance it looks like nudity); full disclosure: I men…

I suggest you take a look at Geometric Deep Learning, but the gist here is that convolutions can be thought of as translation equivariant functions, and pooling operations as permutation invariant combinations, all operating on a graph of as many components as the number of times the operation will be output, each component is composed of the pixels that the operation will act on, so there is local information that i…

Thank you for bringing up GDL. I've been following its developments, and a really great resource is this site: https://geometricdeeplearning.com/

It contains links to the paper and lectures, and the keynote by M. Bronstein is illuminating and discusses the operations on graphs that lead to equivalence to other network topologies and designs: transformer equivalence, and more.

> Keynote: https://www.youtube.com/watch?v=9cxhvQK9ALQ

How are you learning/studying GDL? Would you like someone else to discuss/learn it with?

Re: Do vision transformers see like convolutional neural networks?

#22

Post by 6gvONxR4sf7o »

omarhaneef wrote: post #3

A useful HN feature would be small space to put in a summary, like the abstract: Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. This raises a central question: how are Vision Transformers solving these tasks? Are they acting like con…

Is the idea to not make people click the links before discussing?

Re: Do vision transformers see like convolutional neural networks?

#23

Post by sillysaurusx »

gamegoblin wrote: post #16
sillysaurusx wrote: post #12

Earlier quoted context omitted.

Doubtful. The biggest downside of transformers for vision is how ungodly long they take to produce results. Tesla has to operate in realtime.

In Karpathy’s recent AI day presentation he specifically stated they use transformers. But not on the raw camera input — they use regnets for that. The transformers come higher up the stack: https://youtu.be/j0z4FweCy4M Transformers mentioned on the slide at timestamp 1:00:18.

Thank you!

Re: Do vision transformers see like convolutional neural networks?

#24

Post by PartiallyTyped »

dpflan wrote: post #21
PartiallyTyped wrote: post #18

Earlier quoted context omitted.

I suggest you take a look at Geometric Deep Learning, but the gist here is that convolutions can be thought of as translation equivariant functions, and pooling operations as permutation invariant combinations, all operating on a graph of as many components as the number of times the operation will be output, each component is composed of the pixels that the operation will act on, so there is local information that i…

Thank you for bringing up GDL. I've been following its developments, and a really great resource is this site: https://geometricdeeplearning.com/ It contains links to the paper and lectures, and the keynote by M. Bronstein is illuminating and discusses the operations on graphs that lead to equivalence to other network topologies and designs: transformer equivalence, and more. > Keynote: https://www.youtube.com/watch?…

I believe Bronstein is onto something huge here, with massive implications. Illuminating is the best adjective to describe it, as I watched this keynote:

> https://www.youtube.com/watch?v=w6Pw4MOzMuo

Everything clicked into place and I was given a new language to see the world that combined everything together well beyond the way standard DL is taught:

> we do feature extraction using this function that resembles the receptive fields of the visual cortex and then we project the dense feature representation onto multiple other vectors and pass that through stacked non-linearities, and oh by the way we have myriad of different, seemingly disconnected, architectures that we are not sure why they work, but we call it inductive bias.

> https://geometricdeeplearning.com/

That's my main source, along with the papers that lead up to the proto-book, so pretty much Bronstein's work along with related papers found using `connectedpapers.com`. I don't have an appropriate background so I am grinding through abstract algebra, geometric algebra, will then go into geometry and whatever my supervisor suggests I should read. Sure, I would like to have other people to discuss it, but don't expect much just yet.

Re: Do vision transformers see like convolutional neural networks?

#25

Post by dpflan »

PartiallyTyped wrote: post #24
dpflan wrote: post #21

Earlier quoted context omitted.

Thank you for bringing up GDL. I've been following its developments, and a really great resource is this site: https://geometricdeeplearning.com/ It contains links to the paper and lectures, and the keynote by M. Bronstein is illuminating and discusses the operations on graphs that lead to equivalence to other network topologies and designs: transformer equivalence, and more. > Keynote: https://www.youtube.com/watch?…

I believe Bronstein is onto something huge here, with massive implications. Illuminating is the best adjective to describe it, as I watched this keynote: > https://www.youtube.com/watch?v=w6Pw4MOzMuo Everything clicked into place and I was given a new language to see the world that combined everything together well beyond the way standard DL is taught: > we do feature extraction using this function that resembles the…

I agree, this perspective is very interesting and tames the zoo of architectures through mathematical unification. It is indeed exciting!

Good luck with your studies/learning!

Re: Do vision transformers see like convolutional neural networks?

#26

Post by tandem5000 »

gamegoblin wrote: post #16
sillysaurusx wrote: post #12

Earlier quoted context omitted.

Doubtful. The biggest downside of transformers for vision is how ungodly long they take to produce results. Tesla has to operate in realtime.

In Karpathy’s recent AI day presentation he specifically stated they use transformers. But not on the raw camera input — they use regnets for that. The transformers come higher up the stack: https://youtu.be/j0z4FweCy4M Transformers mentioned on the slide at timestamp 1:00:18.

They use the key-value lookup/routing mechanism from Transformers to predict pixel-wise labels in bird view (lane, car, obstacle, intersection etc.). The motivation here is that some of the predictions may temporarily be occluded, so for predicting these occluded areas it may be particularly helpful to attend to remote regions in the input images which requires long-range dependencies that highly depend on the input itself (e.g. on whether there is an occlusion), which is exactly where the key-value mechanism excels. Not sure they even process past camera frames at this point. They only mention that later in the pipline they have an LSTM-like NN incorporating past camera frames (Schmidhuber will be proud!!).

Edit: A random observation which just occurred to me is that their predictions seem surprisingly temporally unstable. Observe, for example, the lane layout wildly changing while it drives makes a left-turn at the intersection (https://youtu.be/j0z4FweCy4M?t=2608). You can use the comma and period keys to step through the video frame-by-frame.

Re: Do vision transformers see like convolutional neural networks?

#27

Post by codeflo »

stared wrote: post #13

I am much more interested if they fall for the same tricks. For example, if it is easy to fool them with optical illusions, such as innocent images that look racy at the first glance: https://medium.com/@marekkcichy/does-ai-have-a-dirty-mind-to... CW: Even though it does not contain a single explicit picture, it might be considered NSFW (literally - as at the first glance it looks like nudity); full disclosure: I men…

Those pictures are definitely NSFW when viewed at low res/from far away, which is how coworkers typically see your monitor contents. An argument that starts with “Well, technically” is unlikely to carry much weight in a discussion with HR (and probably rightfully so).

Re: Do vision transformers see like convolutional neural networks?

#28

Post by omarhaneef »

codetrotter wrote: post #4
omarhaneef wrote: post #3

A useful HN feature would be small space to put in a summary, like the abstract: Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. This raises a central question: how are Vision Transformers solving these tasks? Are they acting like con…

> A useful HN feature would be small space to put in a summary, like the abstract Interestingly, HN does actually save accompanying text when you submit a link also. It just doesn’t show the text on the website. https://news.ycombinator.com/item?id=28180298 Personally I like it the way that it is. I think showing an accompanying text for links would allow too much for anyone posting a link to “force” everyone to read…

I guess this is really in response to all the other responses as well, but I thought the idea would be:

to help people decide if they want to click the link.

So the title may not be sufficiently informative to let people know if they can understand the article, are interested in it, if it is at the right technical level and so on.

I think you're right that it will be abused in many instances and might not be worth it.

Re: Do vision transformers see like convolutional neural networks?

#29

Post by dexter89_kp3 »

sillysaurusx wrote: post #17
dexter89_kp3 wrote: post #14

Earlier quoted context omitted.

They do. Karpathy spoke about it in on Tesla AI day. They use it for transforming image space to a vector space. See: https://youtu.be/j0z4FweCy4M (timestamp 54.40 onwards)

Can you provide a more specific timestamp? 54.40 doesn't seem to mention anything about transformers, and "onwards" is two hours. I'd be really surprised if they use transformers due to how computationally expensive they are for anything involving vision. EDIT: Found it. 1h: https://www.youtube.com/watch?v=j0z4FweCy4M?t=1h Fascinating. I guess transformers are efficient.

I gave the timestamp where they start talking about the problem they are trying to solve using transformers. As you said it is around 1hr mark

Re: Do vision transformers see like convolutional neural networks?

#30

Post by _untom_ »

stared wrote: post #13

I am much more interested if they fall for the same tricks. For example, if it is easy to fool them with optical illusions, such as innocent images that look racy at the first glance: https://medium.com/@marekkcichy/does-ai-have-a-dirty-mind-to... CW: Even though it does not contain a single explicit picture, it might be considered NSFW (literally - as at the first glance it looks like nudity); full disclosure: I men…

Not sure if this is along the lines of what you're thinking, but we tried looking at this a while ago: https://arxiv.org/abs/2103.14586
Post reply on HN