Playing with Vision Embeddings
prestonbjensen.com
Playing with Vision Embeddings
1–10 of 18 posts
Re: Playing with Vision Embeddings
#2[deleted]
Re: Playing with Vision Embeddings
#3Very nice visualizations, thanks for that!
One thing I still struggle with in my head is how these vision embeddings can then be used to give LLMs eyes.
Because you somehow need a giant training set which describes images in natural language, no? Is that actually how it works, or is there some smart trick so you don't need to pay labellers a bunch of money to look at pictures and describe them.
Re: Playing with Vision Embeddings
#4Very nice visualizations, thanks for that! One thing I still struggle with in my head is how these vision embeddings can then be used to give LLMs eyes. Because you somehow need a giant training set which describes images in natural language, no? Is that actually how it works, or is there some smart trick so you don't need to pay labellers a bunch of money to look at pictures and describe them.
> Because you somehow need a giant training set which describes images in natural language, no?
That's definitely one way - they train a text encoder together with an image encoder on a labelled set of images. WL & 3b1b made a nice video on it: https://www.youtube.com/watch?v=iv-5mZ_9CPY
Re: Playing with Vision Embeddings
#5Very nice visualizations, thanks for that! One thing I still struggle with in my head is how these vision embeddings can then be used to give LLMs eyes. Because you somehow need a giant training set which describes images in natural language, no? Is that actually how it works, or is there some smart trick so you don't need to pay labellers a bunch of money to look at pictures and describe them.
> Because you somehow need a giant training set which describes images in natural language, no? That's definitely one way - they train a text encoder together with an image encoder on a labelled set of images. WL & 3b1b made a nice video on it: https://www.youtube.com/watch?v=iv-5mZ_9CPY
Thanks I'll check out that video
Re: Playing with Vision Embeddings
#6This reminds me of GANs and generating directly from a latent vector, as well as exploring them. I wonder how many of the original tricks used with GANs (like extrapolation stuff, optimizing for the latent vector to get a result etc) are easy to replicate with it.
Re: Playing with Vision Embeddings
#7Beautiful illustrations
I find, 'Playing' is just the free and motivated version of 'exploration'.
One thought on your nicely illustrated "key observation [is] that neural networks tend to place features along directions": my guess is that the neural net was TOLD to behave that way by choosing e.g. Cosine Loss?
Re: Playing with Vision Embeddings
#8For some reason, the uncanniness of the feature pictures are deeply unsettling for me. It just stirs intense unease. A bit amusing, to be honest.
Re: Playing with Vision Embeddings
#9[deleted]
Re: Playing with Vision Embeddings
#10[flagged]