Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MiniGPT-4

minigpt-4.github.io

Post reply on HN
Print view View original

181–190 of 337 posts

Re: MiniGPT-4

#181

Post by Tepix »

HarHarVeryFunny wrote: post #77
Uehreka wrote: post #61

Earlier quoted context omitted.

The word for this is “trademark infringement”. You are specifically not allowed to capitalize on the marketing of another entity’s product to bolster yours by implying through your name that you are somehow related. This is why “DALL-E Mini” had to change their name to craiyon.

It's also just (deliberately) misleading. It's based on the 13B Vicuna/Llama model, not 175B GPT-3 or 1T GPT-4. There is zero justification for calling it MiniGPT-4. A more honest name would be Visual-Vicuna or Son-of-BLIP.

> 1T GPT-4

The number of parameters used for GPT-4 is unknown.

Re: MiniGPT-4

#182

Post by thewarrior »

sho wrote: post #14
152334H wrote: post #6

On a technical level, they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs. But the results are pretty amazing. It completely knocks Openflamingo && even the original blip2 models out of the park. And best of all, it arrived before OpenAI's GPT-4 Image Modality did. Real win for Open Sourc…

> they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs Oh yes. Simple! Jesus, this ML stuff makes a humble web dev like myself feel like a dog trying to read Tolstoy.

I've only been reading ML stuff for a few months and I kind of understand what it's saying. This stuff isn't as complex as its made out to be.

It's just a bunch of black boxes AKA "pure functions".

BLIP2's ViT-L+Q-former AKA

    //I give you a picture of a plate of lobster it will say "A plate of lobster".

    getTextFromImage(image) -> Text
Vicuna-13B AKA

    //I give you a prompt and you return completion ChatGPT style
     getCompletionFromPrompt(text) -> Text
We want to take the output of the first one and then feed in a prompt to the LLM (Vicuna) that will help answer a question about the image. However the datatypes don't match. Lets add in a mapper.

    getAnswerToQuestion(image, question) -> answer 
        text = getTextFromImage(image)
        prompt = mapTextToPrompt(text)
        return getCompletionForPrompt(prompt)
Now where did this mapTextToPrompt come from ?

This is the magic of ML. We can just "learn" this function from data. And they plugged in a "simple" layer and learned it from a few examples of (image , question) -> answer. This is what frameworks like Keras, Pytorch allow you to do. You can wire up these black boxes with some intermediate layers and pass in a bunch of data and voila you have a new model. This is called differentiable programming.

The thing is you don't need to convert to text and then map back into numbers to feed into the LLM. You skip that and use the numbers it outputs and multiply directly with an intermediate matrix.

    getAnswerToQuestion(image, question) -> answer 
        text = getEmbeddingFromImage(image)
        embedding = mapEmbeddingToInputEmbeddingForLLM(text)
        return getCompletionForEmbedding(embedding)
Congratulations you now understood that sentence.

Re: MiniGPT-4

#183

Post by cs702 »

visarga wrote: post #178
cs702 wrote: post #142

Earlier quoted context omitted.

Good point. Didn't think of that. It's a plausible explanation here, because the dimensionality of the spaces is so different, 5120 vs 768. Not surprisingly, the trained weight matrix has rank 768: it's using every feature in the lower-dimensional space. Still, it's kind of shocking that it works so well! I'd be curious to see if the learned weight matrix ends up being full-rank (or close to full-rank) if both spaces…

They would have full-rank because all the embedding space is used. There are no unused large pockets.

The weight matrix's rank would decrease for each feature in the target space that cannot be expressed as as a linear combination of features in the input space (plus a bias). For example, if the target space has a feature representing a non-visual quality like "smelliness," it would not be expressible as a linear combination of features representing visual attributes like "redness," "blueness," and "greenness," etc. in the input space.

If both spaces have the same dimensionality, the learned weight matrix would be full-rank only if every feature in the target space is expressible as a linear combination of features in the input space (plus a bias). Which brings me back to my original question: WHY would that be the case when the two models are trained independently on data that is so different?

Re: MiniGPT-4

#184

Post by visarga »

sho wrote: post #14
152334H wrote: post #6

On a technical level, they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs. But the results are pretty amazing. It completely knocks Openflamingo && even the original blip2 models out of the park. And best of all, it arrived before OpenAI's GPT-4 Image Modality did. Real win for Open Sourc…

> they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs Oh yes. Simple! Jesus, this ML stuff makes a humble web dev like myself feel like a dog trying to read Tolstoy.

> take BLIP2's ViT-L+Q-former

This thing takes an image and creates a representation matrix.

> connect it to Vicuna-13B with a linear layer

Vicuna is an open LLM, pretty good quality, not as good as GPT3.5 though.

This is the beautiful part - a mere multiplication is enough to convert the image tensor to text tensor. One freaking line of code, and a simple one.

> and train just the tiny layer on some datasets of image-text pairs

You then get a shitload of image-text pairs and train the model to describe the images in text. But keep both the image and text model frozen. Is that hard? No, just flip a flag. So this "linear projection layer" (a matrix multiplication) is the only learned part. That means it takes less time to train, needs fewer examples and requires less memory.

Training the image and text models was much more difficult. But here we don't train these models, they use them as ready-made parts. It's a hack on top of two unrelated models, so it is cheap.

In the end the finishing touches - they label 3500 high quality image-text pairs, and fine-tune on them. Now the model becomes truly amazing. It has broad visual intelligence, and scooped OpenAI who didn't release Image GPT-4 in the APIs yet.

The important lesson to take is that unrelated models can be composed together with a bit of extra training for the glue model. And that open AI is just as powerful as "Open"AI sometimes. It's breathing down their necks, just one step behind. This model is also significant for applications - it can power many automations in a flexible way.

Re: MiniGPT-4

#185

Post by whywhywhywhy »

mewpmewp2 wrote: post #53
redox99 wrote: post #46

Earlier quoted context omitted.

It's pretty simple actually. Get a 3090 or 4090. Forget about AMD.

Should I get a gaming PC with 4090 if I want to get into personal side projects in ML and also games? Do I need dualboot? Or is Windows good?

Save some money and go 3090, same vram, speed difference probably isn't worth the premium for the 4090. Then upgrade when the rumored 5090 generational leap happens.

Re: MiniGPT-4

#186

Post by Imnimo »

I'm pretty surprised that the frozen encoder has enough information in its representations to do the "design the website from a hand-written whiteboard mock-up" task, and that that information can be effectively accessed by the language model with just a small projection layer. I really would have guessed that you needed to be training the image encoder simultaneously.

Re: MiniGPT-4

#187

Post by burkaman »

m348e912 wrote: post #45
xnzakg wrote: post #31

The ramen example is kind of hilarious. Wonder if it would make more sense with a bigger model.

Ok I'll bite. What makes the Raman example so funny?

The recipe says to make a broth, cook the noodles in the broth, then throw the broth away and add the noodles to a pan with meat. You'll end up with burned crispy noodles and meat in a dry bowl.

Re: MiniGPT-4

#188

Post by teacpde »

iandanforth wrote: post #163
twelvechairs wrote: post #42

Earlier quoted context omitted.

Its poor form to be calling it 'Open' AI. So I guess its swings and roundabouts. Everyone is leeching where they can.

To be fair they were open when that name was picked and it looks like they may be trying to transition to just 'ai.com'.

Wow, that must be an expensive domain name.

Re: MiniGPT-4

#189

Post by Tepix »

22c wrote: post #137
valine wrote: post #64

Earlier quoted context omitted.

Cheap is relative I suppose. I’m running Vicuna 13b 16f locally and it needs 26GB of VRAM, which won’t even fit on a single RTX 4090. The next gen RTX Titan might have enough vram but that won’t come cheap. I’m expecting a price point above $2500.

I'm not sure if it's the point GP is trying to make, but I would like to see GPUs with extra VRAM that don't have the extra compute. eg. similar performance of a 4070Ti but with 24GB or 32GB of VRAM. I don't see a really good reason why OEMs couldn't do that now, in the past there have been OEM cards that have more VRAM than the reference design. I'm sure there's an appetite for cards like that for people who don't w…

I'm fairly sure that NVIDIA is making sure that consumer cards are no good alternative to their $10000 80GB VRAM A100 cards.

Re: MiniGPT-4

#190

Post by causi »

astrange wrote: post #87
causi wrote: post #21

Earlier quoted context omitted.

I never know what to expect anymore. We live in a world where computers can describe paintings and write sonnets about them but a half-trillion dollar car company can't figure out how to parallel park with eight cameras.

That's a well known result from the last AI hype era. https://en.wikipedia.org/wiki/Moravec%27s_paradox Although, I'm not surprised AI can describe paintings, but I'm still pretty surprised it can generate them. How'd we get that to work!?

Yeah, and I'm still completely lost as to why resolution is such a limiting factor. If you know you're drawing a soccer ball why is a 512x512 soccer ball so much easier than a 1024x1024 soccer ball?
Post reply on HN