The ramen example is kind of hilarious. Wonder if it would make more sense with a bigger model.
MiniGPT-4
31–40 of 337 posts
Re: MiniGPT-4
#32On a technical level, they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs. But the results are pretty amazing. It completely knocks Openflamingo && even the original blip2 models out of the park. And best of all, it arrived before OpenAI's GPT-4 Image Modality did. Real win for Open Sourc…
With fp8, would 4GB be enough or is 6GB more like it?
Re: MiniGPT-4
#33Earlier quoted context omitted.
What did you expect?
I never know what to expect anymore. We live in a world where computers can describe paintings and write sonnets about them but a half-trillion dollar car company can't figure out how to parallel park with eight cameras.
That's because the decided they do not need lidar.
Re: MiniGPT-4
#34On a technical level, they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs. But the results are pretty amazing. It completely knocks Openflamingo && even the original blip2 models out of the park. And best of all, it arrived before OpenAI's GPT-4 Image Modality did. Real win for Open Sourc…
It sounds like a BLIP2 with an extra linear layer for finetuning (or aligning the Q-former with a new LLM?). What makes it more powerful than BLIP2?
It's better because
1. it's using vicuna as a base.
2. It has a pretty high quality fine-tuning dataset. I initially missed this, and it's a very important advantage.
3. (speculatively) it doesn't collapse to extremely short responses (which BLIP2 and other models trained on image-text caption pairs) because of how small/simple the adapter is.
I was interested in training a BLIP2-LLaMA model before this, and I might still do it just to test (3).
Re: MiniGPT-4
#35Earlier quoted context omitted.
I never know what to expect anymore. We live in a world where computers can describe paintings and write sonnets about them but a half-trillion dollar car company can't figure out how to parallel park with eight cameras.
That's because the decided they do not need lidar.
Couldn't afford*
Re: MiniGPT-4
#36On a technical level, they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs. But the results are pretty amazing. It completely knocks Openflamingo && even the original blip2 models out of the park. And best of all, it arrived before OpenAI's GPT-4 Image Modality did. Real win for Open Sourc…
Thanks for a useful comment. Do you reckon the 4bit quantized Vicuna just won't do here? https://huggingface.co/anon8231489123/vicuna-13b-GPTQ-4bit-1... I think with this everything OpenAI demonstrated ~5 weeks ago has been recreated by actually-open AI. Even if it runs much much slower on prosumer hardware and with less good results at least it is de-magicked.
It'll work! I just haven't touched any of the 4bit stuff myself, so I don't personally know how to add it. Great low-hanging fruit for anyone else to take on.
Re: MiniGPT-4
#37Giving this a quick spin and I like what I'm seeing. I gave it a picture of Paolo Veronese's Lament over the Dead Christ [0] and asked what was going on in the background. > The painting depicts the scene of the crucifixion of Jesus Christ. In the foreground, there is a group of people standing around the cross, including Mary, the mother of Jesus, and the two thieves who were crucified with him. In the background, t…
What did you expect?
A 14-line poem with a consistent rhyme scheme and meter. Perhaps my request should have been more specific.
Re: MiniGPT-4
#38On a technical level, they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs. But the results are pretty amazing. It completely knocks Openflamingo && even the original blip2 models out of the park. And best of all, it arrived before OpenAI's GPT-4 Image Modality did. Real win for Open Sourc…
> they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs Oh yes. Simple! Jesus, this ML stuff makes a humble web dev like myself feel like a dog trying to read Tolstoy.
Seriously, ChatGPT was the thing that gave me a foothold into the AI/machine learning world... because it gave me hope that a mere mortal can achieve something reasonable with this tech without a crazy amount of work and educational background.
Re: MiniGPT-4
#39Earlier quoted context omitted.
> they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs Oh yes. Simple! Jesus, this ML stuff makes a humble web dev like myself feel like a dog trying to read Tolstoy.
Seriously, ChatGPT was the thing that gave me a foothold into the AI/machine learning world... because it gave me hope that a mere mortal can achieve something reasonable with this tech without a crazy amount of work and educational background.
I don't have much experience myself. I only started ~10 months ago -- just a month or two before Stable Diffusion.
You just have to do it every day. It's fun!
Re: MiniGPT-4
#40Giving this a quick spin and I like what I'm seeing. I gave it a picture of Paolo Veronese's Lament over the Dead Christ [0] and asked what was going on in the background. > The painting depicts the scene of the crucifixion of Jesus Christ. In the foreground, there is a group of people standing around the cross, including Mary, the mother of Jesus, and the two thieves who were crucified with him. In the background, t…
How about asking it to describe a Robert Williams painting?
Here's a relatively easy one: [1]
[1] - http://www.illustrationartgallery.com/acatalog/WilliamsTimmy...
