Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MiniGPT-4

minigpt-4.github.io

Post reply on HN
Print view View original

191–200 of 337 posts

Re: MiniGPT-4

#191

Post by kolinko »

HopenHeyHi wrote: post #20
152334H wrote: post #6

On a technical level, they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs. But the results are pretty amazing. It completely knocks Openflamingo && even the original blip2 models out of the park. And best of all, it arrived before OpenAI's GPT-4 Image Modality did. Real win for Open Sourc…

Thanks for a useful comment. Do you reckon the 4bit quantized Vicuna just won't do here? https://huggingface.co/anon8231489123/vicuna-13b-GPTQ-4bit-1... I think with this everything OpenAI demonstrated ~5 weeks ago has been recreated by actually-open AI. Even if it runs much much slower on prosumer hardware and with less good results at least it is de-magicked.

Open AI still hasn't exactly reached the level of gpt3.5. GPT-4 is way ahead of anything.

Re: MiniGPT-4

#192

Post by HarHarVeryFunny »

Tepix wrote: post #181
HarHarVeryFunny wrote: post #77

Earlier quoted context omitted.

It's also just (deliberately) misleading. It's based on the 13B Vicuna/Llama model, not 175B GPT-3 or 1T GPT-4. There is zero justification for calling it MiniGPT-4. A more honest name would be Visual-Vicuna or Son-of-BLIP.

> 1T GPT-4 The number of parameters used for GPT-4 is unknown.

I got the 1T GPT-4 number from here - this is the video that goes with the Microsoft "Sparks of AGI" paper, by a Microsoft researcher that had early access to GPT-4 as part of their relationship with OpenAI.

https://www.youtube.com/watch?v=qbIk7-JPB2c

Re: MiniGPT-4

#193

Post by MattPalmer1086 »

pc86 wrote: post #136
kerkeslager wrote: post #103

Earlier quoted context omitted.

And the code GPT gives you won't work, much like a lot of web dev? ;P

Maybe it's because of how I use it, but the code ChatGPT gives me has always been super helpful and 99% correct. But, we have a policy at work not to use it for work product so I have to spend time changing enough of it where it's different, and I'm never copy/pasting anything. Enough changes to the structure and variables to make it sufficiently different that it can't be considered pasting company data into GPT, as…

Would you say that this satisfies the spirit of the company policy? Or is it a bit of a hack to get around it?

I ask because we are about to produce a similar policy at work. We can see the advantages of it, but likewise, we can't have company data held in their systems.

Re: MiniGPT-4

#194

Post by akiselev »

craigching wrote: post #173
ttul wrote: post #158

Earlier quoted context omitted.

The thing is, machine learning sorta requires a few math prerequisites: linear algebra, differential equations, and to some degree vector calculus. Most web developers don’t have this background.

If you want to understand the theory, that's true. If you want to develop an intuitive understanding without having to understand all the nuts and bolts (and I understand that can be a big ask for how some people learn/understand), give this a try: https://karpathy.ai/zero-to-hero.html

The irony is Karpathy presents the limit/epsilon definition of derivatives in the first half hour (quite well IMO and he never actually says “epsilon”) which is very much a nuts and bolts kind of explanation in calculus.

That said, when most people say differential equations they’re usually thinking of analytical solutions which is very much not necessary for practical ML.

Re: MiniGPT-4

#196

Post by visarga »

vlovic wrote: post #125

Do I understand this correctly: they just took Blip2 and replaced the LLM with Vicuna, and to do that they just added a single linear layer to translate between frozen vision encoder and (frozen) Vicuna? Additionally, and importantly, they manually create a high quality dataset for finetuning their model. If that is the case, then this is really a very, very simple paper. But I guess simple things can lead to great i…

Yes, model composability magic.

Re: MiniGPT-4

#197

Post by visarga »

jokoon wrote: post #169

Are there more university/research centric resource of communities regarding AI, I mean things on a more epistemic and higher level, that disregards current ML techniques and implementations? I have no problem when there is a lot of money spent to implement things that can be used and tested, but I wish I could read about the next step of AI, something that involves science and not software engineering. I've yet to r…

If the engineering part is of secondary importance for you, then at least remember about the dataset. It is in the composition and quality of the training data that all the skills find their origin.

Most of our discussions are about model size, but few about dataset. Yet all the scaling laws hint at the great usefulness of more data. Sometimes even little data can have a great impact in the fine-tuning phase. In the end it is the training data that transforms a random init into the model.

Re: MiniGPT-4

#198

Post by wg0 »

pmoriarty wrote: post #40
wsgeorge wrote: post #3

Giving this a quick spin and I like what I'm seeing. I gave it a picture of Paolo Veronese's Lament over the Dead Christ [0] and asked what was going on in the background. > The painting depicts the scene of the crucifixion of Jesus Christ. In the foreground, there is a group of people standing around the cross, including Mary, the mother of Jesus, and the two thieves who were crucified with him. In the background, t…

How about asking it to describe a Robert Williams painting? Here's a relatively easy one: [1] [1] - http://www.illustrationartgallery.com/acatalog/WilliamsTimmy...

I tried that and here's what it has to say:

> This painting appears to be a depiction of a child running down a dirt road with a kite in hand. The child is wearing a yellow shirt and blue shorts, and the background is filled with various colors and shapes. The overall mood of the painting is playful and carefree.

What's amazing is that it has correctly described the scene including details such as:

  - A boy
  - On a dirt road.
  - Having a Kite in hand.
  - Wearing Yellow shirt and blue shorts.

Re: MiniGPT-4

#199

Post by adam_arthur »

tarruda wrote: post #18

Looking forward to the next generation of cheap GPUs with enough VRAM to run models like Vicuna-13 locally.

A lot of people are running fairly powerful models directly on the CPU these days... seems like inference will not be a GPU exclusive activity going forward. Given that RAM is the main bottleneck at this point, running on CPU seems more practical for most end users

See: https://news.ycombinator.com/item?id=35602234

Re: MiniGPT-4

#200

Post by fpgaminer »

cs702 wrote: post #115
152334H wrote: post #6

On a technical level, they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs. But the results are pretty amazing. It completely knocks Openflamingo && even the original blip2 models out of the park. And best of all, it arrived before OpenAI's GPT-4 Image Modality did. Real win for Open Sourc…

Indeed, really simple. And yes, the results are shockingly good. But what I find most remarkable about this is that the ViT-L+Q-former's hidden states are related by only a linear projection (plus bias) to the Vicuna-13B's token embeddings: emb_in_vicuna_space = emb_in_qformer_space @ W + B These two models are trained independently of each other, on very different data (RGB images vs integer token ids representing s…

BLIP2 is a contrastive Image-Language model. The embeddings from the BLIP2 image model are already both aligned with text, and linear. It should not be a surprise that only a projection is required to translate it to LLaMA's embedding space.
Post reply on HN