Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MiniGPT-4

minigpt-4.github.io

Post reply on HN
Print view View original

251–260 of 337 posts

Re: MiniGPT-4

#251

Post by artificial »

thewarrior wrote: post #182
sho wrote: post #14

Earlier quoted context omitted.

> they're doing something really simple -- take BLIP2's ViT-L+Q-former, connect it to Vicuna-13B with a linear layer, and train just the tiny layer on some datasets of image-text pairs Oh yes. Simple! Jesus, this ML stuff makes a humble web dev like myself feel like a dog trying to read Tolstoy.

I've only been reading ML stuff for a few months and I kind of understand what it's saying. This stuff isn't as complex as its made out to be. It's just a bunch of black boxes AKA "pure functions". BLIP2's ViT-L+Q-former AKA //I give you a picture of a plate of lobster it will say "A plate of lobster". getTextFromImage(image) -> Text Vicuna-13B AKA //I give you a prompt and you return completion ChatGPT style getComp…

Thank you for the insightful breakdown. Cheers!

Re: MiniGPT-4

#252

Post by macrolime »

thewataccount wrote: post #224
macrolime wrote: post #157

Earlier quoted context omitted.

Just to add to the confusion, there's another older RTX 6000 with 24GB ram. This is from an even older generation, same as the GeForce 20 series.

You're kidding? So they called it the RTX 6000, then called it the RTX A6000 for ampere, then back to RTX 6000 for Ada? Why do they do this? Sometimes consumer products are versioned weirdly to mislead customers (like intel cpus) - but these wouldn't even make sense to do that with as they're enterprise cards?

Actually the first one is called Quadro RTX 6000, while the Ada one is just RTX 6000 without "Quadro" in front. Not that it makes the naming make much more sense.

According to GPT-4 the next generation one will be called Galactic Unicorn RTX 6000 :D

Re: MiniGPT-4

#253

Post by artificial »

dharma1 wrote: post #249
mewpmewp2 wrote: post #53

Earlier quoted context omitted.

Should I get a gaming PC with 4090 if I want to get into personal side projects in ML and also games? Do I need dualboot? Or is Windows good?

4090 is amazing, but very large card. 3090 is "good enough" for ML - same 24gb vram - and you can pick them up used for half the price of a new 4090. That's what I did. WSL on windows apparently decent, or native PyTorch, dual boot windows/ubuntu still prob best tho.

Getting CUDA on OpenSUSE was super easy. The Nvidia blob drivers are easy to install and CUDA needs another download and some copy paste. Even Unreal Editor was easier to install than on Windows.

Re: MiniGPT-4

#254

Post by sigmoid10 »

cs702 wrote: post #219
sigmoid10 wrote: post #214

Earlier quoted context omitted.

>somehow they learn to embed different data in feature vectors that are so... similar At it's core, BLIP2 already projects RGB inputs into text token space and Vicuna (or rather LLaMA) uses such tokens as inputs as well as outputs. The only reason why a linear layer is needed at all is because they are not trained at the same time, so you still have to move text embeddings from one space to another. But it should not…

Thanks. Your comment about BLIP2 already projecting RGB inputs into (a different) text token space makes sense to me. See also fpgaminer's comment at https://news.ycombinator.com/item?id=35603246 . However, I don't see how the universal approximation theorem is relevant here. The fact that deep models with sufficient capacity can approximate any function does not imply that two deep models trained independently of ea…

>I don't see how the universal approximation theorem is relevant here. The fact that deep models

The universal approximation is exactly not about deep models. Deep means many layers. But in the most simple (and proven) case, a single hidden layer perceptron is all it needs according to the UAT. Technically it also needs a nonlinear activation function, but you get all sorts of nonlinearities for free downstream anyways in this particular model.

Re: MiniGPT-4

#255

Post by thewarrior »

eternauta3k wrote: post #231
thewarrior wrote: post #182

Earlier quoted context omitted.

I've only been reading ML stuff for a few months and I kind of understand what it's saying. This stuff isn't as complex as its made out to be. It's just a bunch of black boxes AKA "pure functions". BLIP2's ViT-L+Q-former AKA //I give you a picture of a plate of lobster it will say "A plate of lobster". getTextFromImage(image) -> Text Vicuna-13B AKA //I give you a prompt and you return completion ChatGPT style getComp…

Interesting, so the LLM is "just" getting your question plus a normal text description of the image (as vectors)?

At a high level yes.

More precisely - It gets the question After irs passed through a matrix that transforms the text description of the image so the LLM can “understand” it.

It maps from the space of one ML model to the other.

Re: MiniGPT-4

#256

Post by BeefWellington »

lhl wrote: post #245
BeefWellington wrote: post #230

Earlier quoted context omitted.

I'm aware that a 6800 is not RDNA3. You stated broadly: > Current AMD consumer cards have terrible software support and IMO isn't really an option. On Windows you might be able to use SHARK or DirectML ports, but nothing will run out of the box. I was merely sharing that I did not have that same experience that current consumer cards have terrible support.

Sure, and I was merely clarifying that only last-gen architectures work. While I'm glad that SD works for you, if we are getting into it, I think that having no support on current-gen flagship models does equal broadly terrible software support and is more worth highlighting, since otherwise someone might assume they could just go pick one of those up and get a 24GB GPU on the cheap, especially in the context of LLMs…

> I think that having no support on current-gen flagship models does equal broadly terrible software support and is more worth highlighting, since otherwise someone might assume they could just go pick one of those up and get a 24GB GPU on the cheap, especially in the context of LLMs (which is what the OP was asking about).

Absolutely fair and I agree with this part. I started my reply with "FWIW" (For What It's Worth) on purpose.

> For RDNA2, you apparently can get LLMs running, but it requires forking/patching both bitsandbytes and GPTQ: https://rentry.org/eq3hg - and this will be true for any library (eg, can you use accelerate? deepspeed? fastgen? who knows, but certainly no one is testing it and AMD doesn't care if you're not on CDNA).

I haven't tried any of the GPU-based LLMs yet. SD leveraging PyTorch (which seems to have solid ROCm support) worked for me. It will not be faster than NVIDIA for sure but if someone already has a 16GB+ AMD card they may be able to at least play with stuff without needing to purchase an NVIDIA card instead.

Re: MiniGPT-4

#257

Post by boppo1 »

akiselev wrote: post #194
craigching wrote: post #173

Earlier quoted context omitted.

If you want to understand the theory, that's true. If you want to develop an intuitive understanding without having to understand all the nuts and bolts (and I understand that can be a big ask for how some people learn/understand), give this a try: https://karpathy.ai/zero-to-hero.html

The irony is Karpathy presents the limit/epsilon definition of derivatives in the first half hour (quite well IMO and he never actually says “epsilon”) which is very much a nuts and bolts kind of explanation in calculus. That said, when most people say differential equations they’re usually thinking of analytical solutions which is very much not necessary for practical ML.

I dunno, I just watched it and he almost immediately throws out the formality of the delta-epsilon definition and starts using infinitesimals.

Thank god.

Re: MiniGPT-4

#258

Post by boppo1 »

causi wrote: post #26
Tepix wrote: post #12

I think it's poor form that they are taking the GPT-4 name for an unrelated project. After all, the underlying Vicuna is merely a fine-tuned LLaMA. Plus they use the smaller 13B version. The results look interesting, however. Here's hoping that they'll add GTPQ 4bit quantizing so the 65B version of the model can be run on 2x 3090.

Someone needs to write a buyer's guide for GPUs and LLMs. For example, what's the best course of action if don't need to train anything but do want to eventually run whatever model becomes the first local-capable equivalent to ChatGPT? Do you go with Nvidia for the CUDA cores or with AMD for more VRAM? Do you do neither and wait another generation?

3090 or 4090. That's it as far as LLMs go. You can run 13b on a 4080 but 13B sucks compared to 30B. And ROCm is a poorly supported hair-pulling nightmare.

Re: MiniGPT-4

#259

Post by cs702 »

sdenton4 wrote: post #222
cs702 wrote: post #183

Earlier quoted context omitted.

The weight matrix's rank would decrease for each feature in the target space that cannot be expressed as as a linear combination of features in the input space (plus a bias). For example, if the target space has a feature representing a non-visual quality like "smelliness," it would not be expressible as a linear combination of features representing visual attributes like "redness," "blueness," and "greenness," etc.…

A random nxn matrix is full rank... So it's kinda the default: any amount of noise in the embedding is going to result in full-rank transformations. So it's really less-than-full rank which would require an explanation - ie, why does this image representation project into this perfectly isolated subspace of the language representation (or vice versa)? If that happened I would start looking for things like a vocabular…

> But we use cross-modal analogies in language /constantly/ (many smells are associated with things we can see - 'smells like a rose') so you wouldn't expect any clean separations for different modalities...

That's a really good point. Thank you!

Re: MiniGPT-4

#260

Post by cs702 »

sigmoid10 wrote: post #254
cs702 wrote: post #219

Earlier quoted context omitted.

Thanks. Your comment about BLIP2 already projecting RGB inputs into (a different) text token space makes sense to me. See also fpgaminer's comment at https://news.ycombinator.com/item?id=35603246 . However, I don't see how the universal approximation theorem is relevant here. The fact that deep models with sufficient capacity can approximate any function does not imply that two deep models trained independently of ea…

>I don't see how the universal approximation theorem is relevant here. The fact that deep models The universal approximation is exactly not about deep models. Deep means many layers. But in the most simple (and proven) case, a single hidden layer perceptron is all it needs according to the UAT. Technically it also needs a nonlinear activation function, but you get all sorts of nonlinearities for free downstream anywa…

You'd need to increase width (dimensionality) if you make these models shallow.

My point still stands: The fact that models with sufficient capacity can approximate any function does not imply that two models trained independently of each other on different tasks will learn to approximate functions that relate to each other only by a linear transformation.

Post reply on HN