Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

QLoRA: Efficient Finetuning of Quantized LLMs

arxiv.org

Post reply on HN
Print view View original

81–90 of 116 posts

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#81

Post by yyyk »

l33tman wrote: post #71
yyyk wrote: post #62

Earlier quoted context omitted.

>Open models... are still behind GPT-4 by some distance but they are closing in at a rate that leads me to believe there’s not much of a moat there Commercially, this doesn't matter even if true. As I keep reminding here: moats are rarely about raw tech. Moats are much more about integrations, brand ("Nobody got fired for buying X"), access to API/raw compute... If OpenAI have a superior Office 365 integration they'l…

I agree with you on that a good integrator can get successful, but we should not forget here that the underlying code running all of this is in the order of hundreds of lines (see alpaca.cpp for example) in combination with a boatload of money required for training (that is quickly going down), to simplify the argument a bit. It's hardly comparable to the gigantic ecosystem of services and micro-services that your co…

What you refer as 'the underlying code' isn't the real underlying code, the underlying code is the model. Your underlying code is just the runtime, which is commercially the least interesting part (which is why it is/will be/ open sourced). Both the model and the integration are way more interesting commercially. I'm sure the code Microsoft has to integrate their assistant to Windows is way more than hundreds of lines. How long until GNOME/KDE even start on something similar? Probably not before 2030.

Anyway, I treat OpenAI and Microsoft as two sides of the same coin given level of integration between the two. It's arguable the Microsoft has the upper hand here but OpenAI is their main LLM talent. [EDIT: I don't see MS switching backends from a backend they control, especially when performance apparently is adequate enough already and the real cost isn't licensing the code, but Azure, so open source doesn't necessarily have an advantage here.]

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#82

Post by londons_explore »

simonw wrote: post #2

I'm very impressed at the quality of Guanaco 33B, the model that accompanies this paper. You can try it out here: https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi I tried "You are a sentient cheesecake that teaches people SQL, with cheesecake analogies to illustrate different points. Teach me to use count and group by" and got a good result from it: https://twitter.com/simonw/status/1661460336334241794/phot…

Tested the same prompt with GPT-4, and here is the result: https://imgur.com/7y2FiSP

In my opinion, the GPT-4 result is far more informative and less muddled.

Both answers are mostly just regurgitating an SQL tutorial with the objects and column names cheesecake related, so I don't think it's an awfully good test.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#84

Post by amilios »

MacsHeadroom wrote: post #44
andy_xor_andrew wrote: post #15

Hold on. I need someone to explain something to me. The colab notebook shows an example of loading the vanilla, unquantized model "decapoda-research/llama-7b-hf", using the flag "load_in_4bit" to load it as 4bits. When... when did this become possible? My understanding, from playing with these models daily for the past few months, is that quantization of LLaMA-based models is done via this: https://github.com/qwopqwo…

load_in_4bit requires storing a fully unquantized model as well as having enough RAM to load the unquantized model. If you're an enthusiast with 10 models downloaded, do you want that taking up 500GB or 150GB? Do you want to need 64GB of RAM to load a model, or just 16GB? That's the main reason for the popularity of pre-quantization.

Does it? I loaded Guanaco-65B on a 48GB graphics card using the new flag and it loaded just fine. I don't think the unquantized model fits on a 48GB card. So I'm not quite sure if this is true.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#85

Post by api »

yyyk wrote: post #62
api wrote: post #8

Earlier quoted context omitted.

Altman’s push for regulatory capture makes so much sense given how fast this field is going. Open models you can run on regular hardware are still behind GPT-4 by some distance but they are closing in at a rate that leads me to believe there’s not much of a moat there. There’s also a ton of promising work on quantization and pruning and other acceleration and compression techniques to make more powerful models run on…

>Open models... are still behind GPT-4 by some distance but they are closing in at a rate that leads me to believe there’s not much of a moat there Commercially, this doesn't matter even if true. As I keep reminding here: moats are rarely about raw tech. Moats are much more about integrations, brand ("Nobody got fired for buying X"), access to API/raw compute... If OpenAI have a superior Office 365 integration they'l…

I alluded to what you say in the last paragraph: the average user isn't going to play with open source models, but the existence of the open source AI ecosystem will enable more competitors.

There's not going to just be one AI company. There will be thousands and thousands, each addressing different use cases and market niches. In a world where OpenAI has a powerful technological moat, all of these companies would end up having to pay rent to OpenAI. In a world with a strong open source AI ecosystem that's not the case. They can take open source models and even train them themselves and refine them for specific use cases.

Winner take all dynamics in general are overstated. They exist in a few niches but not most. How many networking, database, file sync, cloud, gaming, banking, or hosting companies are there? There's even been markets that once looked winner take all like social media that have recently experienced a flurry of diversification.

Edit: there's one more reason I'm not sure moats are strong in AI: AI can write code and can process "messy" inputs. One of the thing that strengthens moats built around integrations and such is that the difficulty of doing the integration is part of the barrier. Integrations are frankly annoying and labor intensive to create. With AI you can just tell it to integrate in natural language and schlep messy imperfect data into it. That makes integrations significantly less labor intensive, making it easier for a competitor to pop up and add them very easily and quickly.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#86

Post by anentropic »

andy_xor_andrew wrote: post #15

Hold on. I need someone to explain something to me. The colab notebook shows an example of loading the vanilla, unquantized model "decapoda-research/llama-7b-hf", using the flag "load_in_4bit" to load it as 4bits. When... when did this become possible? My understanding, from playing with these models daily for the past few months, is that quantization of LLaMA-based models is done via this: https://github.com/qwopqwo…

Announced yesterday it seems https://huggingface.co/blog/4bit-transformers-bitsandbytes

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#87

Post by amilios »

dharma1 wrote: post #70
gangwolf wrote: post #50

Earlier quoted context omitted.

GPT-4 ELI5: - 4-bit Quantization: Imagine you have a box of 16 different colored crayons. But you realize that you can draw almost the same picture using only 4 colors. That's what quantization does. It reduces the number of different "colors" (or numbers) that the model uses to represent its knowledge, which saves a lot of space. In this case, they used a special kind of 4-bit quantization, which means they only use…

Better than most humans would have come up with! How does this not have some kind of a model of the world, with real world analogies like this?

It probably does have some sort of simplified internal model of the world, who's to say it doesn't?

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#88

Post by api »

akiselev wrote: post #43
sanxiyn wrote: post #33

Earlier quoted context omitted.

> It's only a matter of time before people really crack distributed training algorithms that can be run in a less organized swarm configuration. Why do you think this? This continually failed, and seems extremely unlikely to me. Barring surprising breakthrough, there is inherent communication complexity, and physical limit to communication bandwidth.

Yeah the network bandwidth is insane. Each A100 in an 8x A100 (80GB) pod [1] has its own 200 gigabit NIC ! Including storage that's nine 200gbps interfaces for almost two terabit of total internal bandwidth. The newer H100s each have a 400gbit NIC. [1] https://shop.lambdalabs.com/deep-learning/servers/hyperplane...

The assumption I'm questioning is that it will always take a rig like that to train large models. I don't see anything inherent in the math or nature of these things that forbids a wider, more heterogenous, less synchronized, less orderly training path that involves basically a mob of people training and merging results. We just haven't really figured it out yet.

I actually see a little bit of this happening on Huggingface with people creating variations and "remixes" of generative models like Stable Diffusion and trying to one-up each other or make models to do esoteric things like render everything looking like anime. You're not going to get to the next frontier model with those methods but it shows that the interest exists and a flourishing ecosystem is forming. Now give that ecosystem new methods that are more powerful.

People with more money can obviously buy or rent more hardware. The question is whether that advantage will stay as meaningful as it is today forever.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#89

Post by gumballindie »

CuriouslyC wrote: post #40
gumballindie wrote: post #35

Earlier quoted context omitted.

So that means a massively distributed model training network with cryptocurrency like incentives is incoming? Where and how to begin? This could free up companies such as openai and potentially lead to the first agi. (mentioning crypto because that will motivate switch hordes of miners that already have the gpu power available)

We already have exactly that for stable diffusion with Civitai.com. People have published a variety of LoRAs for different subjects just as you describe. The local LLM community is very much following the lead of the stable diffusion community in terms of how it's organizing, so I expect that we'll see a proliferation of domain LoRAs being published on an aggregator for LLM stuff before too long.

I don't think the two concepts are similar. I see no incentive for people to train for civitai and find no particular use for the generated content.

Edit: actually, some of that content looks suspicious.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#90

Post by yyyk »

api wrote: post #85
yyyk wrote: post #62

Earlier quoted context omitted.

>Open models... are still behind GPT-4 by some distance but they are closing in at a rate that leads me to believe there’s not much of a moat there Commercially, this doesn't matter even if true. As I keep reminding here: moats are rarely about raw tech. Moats are much more about integrations, brand ("Nobody got fired for buying X"), access to API/raw compute... If OpenAI have a superior Office 365 integration they'l…

I alluded to what you say in the last paragraph: the average user isn't going to play with open source models, but the existence of the open source AI ecosystem will enable more competitors. There's not going to just be one AI company. There will be thousands and thousands, each addressing different use cases and market niches. In a world where OpenAI has a powerful technological moat, all of these companies would en…

I agree OpenAI won't and can't have anywhere near a monopoly, excluding very unlikely legal shenanigans. The field is too approachable, and ultimately, to pinch into their market one doesn't need GPT-4 level performance. Besides, apparently OpenAI can't scale compute fast enough, so users will end up by default requiring other providers. There's a very good chance of OpenAI/Microsoft being major players though IMHO.

I can however see a possible future where open source is not going to have any significant impact on LLMs, say like Desktop Linux. Either because it gets stuck in a technical realm and doesn't make anything too approachable to ordinary users, or because it lacks the necessary integrations, or because developers get stuck arguing about the license (raw LLAMA not being good enough due to the non-commercial requirement), or because a moral panic ("4chanGPT is radicalizing people!") leads to a form of legal restrictions that makes open source efforts difficult to sustain. This doesn't have to be, so long as the hacker community can avoid falling into complacency.

On integrations, you're thinking about input, but there are still significant challenges there, the output step, API keys, rate limits, various crazy API corners, certifications... LLMs will help, but I expect integration to still be annoying.

For example, the Microsoft example where the Assistant changes the system to Dark Mode. You can't use LLM messy input to get that output on a generic level (and if you could, that would risk the LLM as an attack vector). You might be able to use a software development LLM to help write the code to do that specific thing and make it available to the product LLM, but ultimately that's a generic software productivity speedup - which also 'helps' those writing the API to make it more complicated and do more stuff we'll need to implement...

Post reply on HN