Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do vision transformers see like convolutional neural networks?

arxiv.org

Post reply on HN
Print view View original

31–40 of 45 posts

Re: Do vision transformers see like convolutional neural networks?

#31

Post by im3w1l »

abalaji wrote: post #10
mrfusion wrote: post #8

Offtopic sort of, but does anyone know if folks are working on combining vision and natural language in one model? I think that could wield some interesting results.

yeah there has definitely been work done in that space: it’s called multi-modal models not sure if this is the latest work but here’s some results from Google’s AI Blog https://ai.googleblog.com/2017/06/multimodel-multi-task-mach...

What would be really cool is neural networks with routing. Like circuit switching or packet switching. No idea how you would train such a beast though.

Like imagine the vision part making a phonecall to the natural language part to ask it for help with something.

Re: Do vision transformers see like convolutional neural networks?

#32

Post by spullara »

stared wrote: post #13

I am much more interested if they fall for the same tricks. For example, if it is easy to fool them with optical illusions, such as innocent images that look racy at the first glance: https://medium.com/@marekkcichy/does-ai-have-a-dirty-mind-to... CW: Even though it does not contain a single explicit picture, it might be considered NSFW (literally - as at the first glance it looks like nudity); full disclosure: I men…

Since it easy to fool people with optical illusions I doubt that you will be able to train a computer to not be fooled by optical illusions.

Re: Do vision transformers see like convolutional neural networks?

#33

Post by stared »

spullara wrote: post #32
stared wrote: post #13

I am much more interested if they fall for the same tricks. For example, if it is easy to fool them with optical illusions, such as innocent images that look racy at the first glance: https://medium.com/@marekkcichy/does-ai-have-a-dirty-mind-to... CW: Even though it does not contain a single explicit picture, it might be considered NSFW (literally - as at the first glance it looks like nudity); full disclosure: I men…

Since it easy to fool people with optical illusions I doubt that you will be able to train a computer to not be fooled by optical illusions.

It fools humans only at the first glance. A few seconds later we make a correct assessement.

Typical CNNs miss this second stage.

Re: Do vision transformers see like convolutional neural networks?

#34

Post by TylerLives »

im3w1l wrote: post #31
abalaji wrote: post #10

Earlier quoted context omitted.

yeah there has definitely been work done in that space: it’s called multi-modal models not sure if this is the latest work but here’s some results from Google’s AI Blog https://ai.googleblog.com/2017/06/multimodel-multi-task-mach...

What would be really cool is neural networks with routing. Like circuit switching or packet switching. No idea how you would train such a beast though. Like imagine the vision part making a phonecall to the natural language part to ask it for help with something.

Sounds like The Society of Mind - https://en.m.wikipedia.org/wiki/Society_of_Mind

Re: Do vision transformers see like convolutional neural networks?

#35

Post by codeflo »

codeflo wrote: post #27
stared wrote: post #13

I am much more interested if they fall for the same tricks. For example, if it is easy to fool them with optical illusions, such as innocent images that look racy at the first glance: https://medium.com/@marekkcichy/does-ai-have-a-dirty-mind-to... CW: Even though it does not contain a single explicit picture, it might be considered NSFW (literally - as at the first glance it looks like nudity); full disclosure: I men…

Those pictures are definitely NSFW when viewed at low res/from far away, which is how coworkers typically see your monitor contents. An argument that starts with “Well, technically” is unlikely to carry much weight in a discussion with HR (and probably rightfully so).

Counter argument in addition to downvote, please?

Re: Do vision transformers see like convolutional neural networks?

#36

Post by codetrotter »

sillysaurusx wrote: post #11
mrfusion wrote: post #8

Offtopic sort of, but does anyone know if folks are working on combining vision and natural language in one model? I think that could wield some interesting results.

https://github.com/openai/CLIP The results are quite interesting: https://www.reddit.com/r/Art/comments/p866wv/deep_dive_meai_...

And here is a short guide and a link to a Google Collab notebook that anyone can use to create their own AI-powered art using VQGAN+CLIP: https://sourceful.us/doc/935/introduction-to-vqganclip

Re: Do vision transformers see like convolutional neural networks?

#37

Post by wittenator »

im3w1l wrote: post #31
abalaji wrote: post #10

Earlier quoted context omitted.

yeah there has definitely been work done in that space: it’s called multi-modal models not sure if this is the latest work but here’s some results from Google’s AI Blog https://ai.googleblog.com/2017/06/multimodel-multi-task-mach...

What would be really cool is neural networks with routing. Like circuit switching or packet switching. No idea how you would train such a beast though. Like imagine the vision part making a phonecall to the natural language part to ask it for help with something.

Capsule networks have a routing algorithm as far as I know

Re: Do vision transformers see like convolutional neural networks?

#38

Post by heavyset_go »

stared wrote: post #33
spullara wrote: post #32

Earlier quoted context omitted.

Since it easy to fool people with optical illusions I doubt that you will be able to train a computer to not be fooled by optical illusions.

It fools humans only at the first glance. A few seconds later we make a correct assessement. Typical CNNs miss this second stage.

And the humor of the images comes from our initial expectations and how different they are from our actual understanding of what we're seeing.

Re: Do vision transformers see like convolutional neural networks?

#39

Post by gugagore »

codeflo wrote: post #27
stared wrote: post #13

I am much more interested if they fall for the same tricks. For example, if it is easy to fool them with optical illusions, such as innocent images that look racy at the first glance: https://medium.com/@marekkcichy/does-ai-have-a-dirty-mind-to... CW: Even though it does not contain a single explicit picture, it might be considered NSFW (literally - as at the first glance it looks like nudity); full disclosure: I men…

Those pictures are definitely NSFW when viewed at low res/from far away, which is how coworkers typically see your monitor contents. An argument that starts with “Well, technically” is unlikely to carry much weight in a discussion with HR (and probably rightfully so).

I think the scientific point here is that visual processing is not a one-shot process. Tasked with object detection, some scenes demand more careful processing and more computation.

Almost all neural network architectures process a given input size in the same amount of time, and some applications and datasets would benefit from an "anytime" approach, where the output is gradually refined given more time.

I understand the point you are making, but it's kind of irrelevant. The task is to produce an answer for the image at the given resolution. It is an accident and coincidence that the neural network produces an answer that is arguably correct for a blurrier version of the image.

Re: Do vision transformers see like convolutional neural networks?

#40

Post by kaibee »

PartiallyTyped wrote: post #18
stared wrote: post #13

I am much more interested if they fall for the same tricks. For example, if it is easy to fool them with optical illusions, such as innocent images that look racy at the first glance: https://medium.com/@marekkcichy/does-ai-have-a-dirty-mind-to... CW: Even though it does not contain a single explicit picture, it might be considered NSFW (literally - as at the first glance it looks like nudity); full disclosure: I men…

I suggest you take a look at Geometric Deep Learning, but the gist here is that convolutions can be thought of as translation equivariant functions, and pooling operations as permutation invariant combinations, all operating on a graph of as many components as the number of times the operation will be output, each component is composed of the pixels that the operation will act on, so there is local information that i…

> I suggest you take a look at Geometric Deep Learning, but the gist here is that convolutions can be thought of as translation equivariant functions, and pooling operations as permutation invariant combinations, all operating on a graph of as many components as the number of times the operation will be output, each component is composed of the pixels that the operation will act on, so there is local information that is slowly combined through the layers, then relative positioning can be decoded when the representation is transformed into a dense 1d-vector aka flattening.

Hey just trying to check my understanding, this is what Taco-tron does right? It outputs an image that can be Fourier transformed into a soundwave, which is the flatting to a dense 1d vector? And the construction of that image works because the network was able to learn from examples of existing sounds transformed into images, because the transformation into an image encodes some invariance that biases the learning network to generalize better or something?

I didn't do really great in math in college, but always found deep learning interesting. Not sure if anything I said above makes any sense.

Post reply on HN