Do vision transformers see like convolutional neural networks?
1–10 of 45 posts
Re: Do vision transformers see like convolutional neural networks?
#2It's always nice to see big labs working more towards building an understanding of things instead of just chasing SOTA. On the other hand, I'm not sure there is a lot of actionable findings in here. I guess that's the trade off with these things though....
Re: Do vision transformers see like convolutional neural networks?
#3A useful HN feature would be small space to put in a summary, like the abstract:
Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. This raises a central question: how are Vision Transformers solving these tasks?
Are they acting like convolutional networks, or learning entirely different visual representations? Analyzing the internal representation structure of ViTs and CNNs on image classification benchmarks, we find striking differences between the two architectures, such as ViT having more uniform representations across all layers.
We explore how these differences arise, finding crucial roles played by self-attention, which enables early aggregation of global information, and ViT residual connections, which strongly propagate features from lower to higher layers. We study the ramifications for spatial localization, demonstrating ViTs successfully preserve input spatial information, with noticeable effects from different classification methods. Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with a discussion on connections to new architectures such as the MLP-Mixer.
Re: Do vision transformers see like convolutional neural networks?
#4A useful HN feature would be small space to put in a summary, like the abstract: Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. This raises a central question: how are Vision Transformers solving these tasks? Are they acting like con…
> A useful HN feature would be small space to put in a summary, like the abstract
Interestingly, HN does actually save accompanying text when you submit a link also. It just doesn’t show the text on the website.
https://news.ycombinator.com/item?id=28180298
Personally I like it the way that it is. I think showing an accompanying text for links would allow too much for anyone posting a link to “force” everyone to read their comment on it. Leaving it so that comments must be posted separately in order to be visible in the thread makes it so that useful accompanying comments can float to the top, whereas a useless comment from the submitter sinks to the bottom while still allowing the submitted link to be voted on individually.
Re: Do vision transformers see like convolutional neural networks?
#5A useful HN feature would be small space to put in a summary, like the abstract: Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. This raises a central question: how are Vision Transformers solving these tasks? Are they acting like con…
There is little point in duplicating the first thing you see on the page on HN too.
Re: Do vision transformers see like convolutional neural networks?
#6It makes sense to me that attention would be hugely beneficial for vision tasks. We use contextual clues every day to decide what we’re looking at.
Re: Do vision transformers see like convolutional neural networks?
#7Does tesla use transformers for the auto pilot?
Re: Do vision transformers see like convolutional neural networks?
#8Offtopic sort of, but does anyone know if folks are working on combining vision and natural language in one model? I think that could wield some interesting results.
Re: Do vision transformers see like convolutional neural networks?
#9It makes sense to me that attention would be hugely beneficial for vision tasks. We use contextual clues every day to decide what we’re looking at.
It may make sense, but it also makes no sense. CNNs already have full view of the entire input image. That's how discriminators are able to discriminate in GANs.
We added attention and observed no benefits at all in our GAN experiments.
Re: Do vision transformers see like convolutional neural networks?
#10Offtopic sort of, but does anyone know if folks are working on combining vision and natural language in one model? I think that could wield some interesting results.
yeah there has definitely been work done in that space: it’s called multi-modal models
not sure if this is the latest work but here’s some results from Google’s AI Blog
https://ai.googleblog.com/2017/06/multimodel-multi-task-mach...