Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

QLoRA: Efficient Finetuning of Quantized LLMs

arxiv.org

Post reply on HN
Print view View original

101–110 of 116 posts

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#101

Post by api »

yyyk wrote: post #90
api wrote: post #85

Earlier quoted context omitted.

I alluded to what you say in the last paragraph: the average user isn't going to play with open source models, but the existence of the open source AI ecosystem will enable more competitors. There's not going to just be one AI company. There will be thousands and thousands, each addressing different use cases and market niches. In a world where OpenAI has a powerful technological moat, all of these companies would en…

I agree OpenAI won't and can't have anywhere near a monopoly, excluding very unlikely legal shenanigans. The field is too approachable, and ultimately, to pinch into their market one doesn't need GPT-4 level performance. Besides, apparently OpenAI can't scale compute fast enough, so users will end up by default requiring other providers. There's a very good chance of OpenAI/Microsoft being major players though IMHO.…

I agree that pure open source probably won't cross over into general use. Open source is great at tech but bad at product, mostly because product requires doing a lot of boring polish that programmers hate and thus generally have to be paid to do (I have ranted about this extensively here and elsewhere). Consumers also hate choice these days because they're all in a state of continuous cognitive overload. They want the one thing to do the one thing and stop bothering me.

That being said, Linux and open source have had a gigantic effect on the market. This hasn't been by shipping products directly to consumers but by enabling a ton of startups to get there faster and cheaper. OSS enables a ton of innovation and consumers benefit from that.

I'm arguing that the same thing is going to happen in AI. That's all. AI will get faster, better, and cheaper, and the basic functionality will get commoditized. That will lead to more competition and more variation and make it hard for people like OpenAI to have a monopoly.

I'm sure OpenAI will still exist. They might stay a dominant player. I just don't see a world where they "own the tech" and get to be "the only AI" and charge rent to the entire industry because nothing works without their API. That's a fantasy... unless they can legislate it into existence, which is what I think they're trying to do.

On integrations: what happens when everyone uses AI? One of the most exciting possibilities I see for this technology is to entirely toss out the persnickety concept of the API in favor of LLMs talking to LLMs. Making software interoperate in the conventional way frankly sucks. It's a terrible slog through mud. Imagine if I could just say "hey app I just wrote, meet GitHub! GitHub, please explain to my app what you can do and how to access your capabilities... Now tell GitHub to... now tell other app on my machine to... now tell Amazon to..."

We are going to look back at how we did software interoperability the way we look at programming mainframes with punched cards.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#102

Post by CuriouslyC »

gumballindie wrote: post #89
CuriouslyC wrote: post #40

Earlier quoted context omitted.

We already have exactly that for stable diffusion with Civitai.com. People have published a variety of LoRAs for different subjects just as you describe. The local LLM community is very much following the lead of the stable diffusion community in terms of how it's organizing, so I expect that we'll see a proliferation of domain LoRAs being published on an aggregator for LLM stuff before too long.

I don't think the two concepts are similar. I see no incentive for people to train for civitai and find no particular use for the generated content. Edit: actually, some of that content looks suspicious.

Some of the people releasing popular models/LoRAs on civitai do alright via Patreon in addition to getting a lot of praise from the SD community, that seems like incentive to me.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#103

Post by GaggiX »

imjonse wrote: post #57
ukuina wrote: post #39

Earlier quoted context omitted.

Which is heavier: a pound of feathers, or a great british pound? > Both weights are equal.

So same answer as free ChatGPT, only terser.

Tbh I asked GPT-3.5 and it responds correctly. Do you have an example where it fails?

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#104

Post by yyyk »

api wrote: post #101
yyyk wrote: post #90

Earlier quoted context omitted.

I agree OpenAI won't and can't have anywhere near a monopoly, excluding very unlikely legal shenanigans. The field is too approachable, and ultimately, to pinch into their market one doesn't need GPT-4 level performance. Besides, apparently OpenAI can't scale compute fast enough, so users will end up by default requiring other providers. There's a very good chance of OpenAI/Microsoft being major players though IMHO.…

I agree that pure open source probably won't cross over into general use. Open source is great at tech but bad at product, mostly because product requires doing a lot of boring polish that programmers hate and thus generally have to be paid to do (I have ranted about this extensively here and elsewhere). Consumers also hate choice these days because they're all in a state of continuous cognitive overload. They want t…

Well, I guess we agree here - the basic functionality will be common, Open AI will not have a monopoly (unless lawmakers really screw up), but will likely exist and be a major player. I am not sure about Open Source at all just yet - e.g. it's possible that a panic will lead to restrictions that will make open work rather difficult. I guess we'll see.

>what happens when everyone uses AI? One of the most exciting possibilities I see for this technology is to entirely toss out the persnickety concept of the API in favor of LLMs talking to LLMs.

Hmm. Your idea has lower throughput and higher latency, while having not entirely clear security properties. But really, companies will sacrifice nearly all that in the service of faster shipping, paying less to programmers, and easier interfacing. Just look at the history of latency in user systems from the 1980s till today. The only game breaker here is security.

Hm. We implement an internal API that the LLM must use in order to do anything local, and do the limiting/auditing there. So the external interface is language, but what the LLM can do is either use the limited internal API to do approved actions locally, or speak to other LLMs (which will have their own internal APIs) using the user's context (OpenID token, whatever). There's a slight risk in the security boundary here (e.g. make sure the LLM never impersonates - don't allow it to see the actual user context, make sure each LLM instance only has access to the current user context. Also need to make sure it doesn't tell the other LLMs too much), but I think it's handleable?

Regardless, we aren't there yet, and there are still a few issues here (e.g. I expect Microsoft/Google to always have an advantage in Office 365/GSuite integrations somehow). Until we do, integrations will still have a big effect on the market.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#106

Post by gumballindie »

seydor wrote: post #97
gumballindie wrote: post #89

Earlier quoted context omitted.

I don't think the two concepts are similar. I see no incentive for people to train for civitai and find no particular use for the generated content. Edit: actually, some of that content looks suspicious.

Porn is a major reason that people train loras. The beauty of them is that people can pick and choose multiple loras to build truly custom ai

I think there may be stuff worse than porn lurking around there.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#107

Post by dTal »

MacsHeadroom wrote: post #95
amilios wrote: post #84

Earlier quoted context omitted.

Does it? I loaded Guanaco-65B on a 48GB graphics card using the new flag and it loaded just fine. I don't think the unquantized model fits on a 48GB card. So I'm not quite sure if this is true.

RAM, not VRAM. The model is fully loaded into (regular, system) RAM then quantized while being loaded into VRAM. So you still need enough system RAM (or RAM+Swap) to load the unquantized model.

Why would it need to be loaded fully into system RAM at once? Couldn't it be quantized on the fly, as it's being read from disk?

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#108

Post by MacsHeadroom »

dTal wrote: post #107
MacsHeadroom wrote: post #95

Earlier quoted context omitted.

RAM, not VRAM. The model is fully loaded into (regular, system) RAM then quantized while being loaded into VRAM. So you still need enough system RAM (or RAM+Swap) to load the unquantized model.

Why would it need to be loaded fully into system RAM at once? Couldn't it be quantized on the fly, as it's being read from disk?

I'm not sure it could be for this complex quantization scheme. It's not a simple round to nearest quantization. Even if it is possible, it would require someone to develop that functionality. Currently it does load the entire unquantized model into RAM first.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#109

Post by MacsHeadroom »

ianpurton wrote: post #64
epups wrote: post #52

Do you know which model size can be run with a 3090?

Assuming you have 24GB of VRAM, then you should be able to run something like 7 billion parameters i.e. MPT-7b and quantized perhaps you can get up to 13 billion. Note: I have no practical experience of this, just reading around.

24GB can fit 33B parameter models in 4bit. You only need 4GB to run 7B models.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#110

Post by coolspot »

gangwolf wrote: post #50
holtkam2 wrote: post #47

Can someone help me understand what quantization means in this context, and why it matters?

GPT-4 ELI5: - 4-bit Quantization: Imagine you have a box of 16 different colored crayons. But you realize that you can draw almost the same picture using only 4 colors. That's what quantization does. It reduces the number of different "colors" (or numbers) that the model uses to represent its knowledge, which saves a lot of space. In this case, they used a special kind of 4-bit quantization, which means they only use…

Guanaco 33B ELI5:

> ELI5 ML model quantization

> Here is a simpler explanation of what AI model quantization means: Imagine you have a big box full of marbles, where each marble represents a piece of information. Each marble has a unique color, pattern, and size that tells you something about the information it represents. Now, let’s say you want to organize these marbles so you can easily retrieve the information later on. One way to do this is to group similar-looking marbles together into buckets. Instead of having a separate bucket for every single marble, you can combine marbles that look alike and put them in the same bucket. This is called “quantization” because you are converting many distinct pieces of information (marbles) into a few representative categories (buckets). It makes things easier to manage and access, but there is some loss of detail as well. The same idea applies to AI models. They are trained on huge datasets that contain lots of different kinds of information. But instead of storing every individual datapoint separately, the model can group similar patterns together and represent them with a few basic templates. This allows the model to run faster and take up less space, but it may lead to some errors or misclassifications.

Post reply on HN