Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

QLoRA: Efficient Finetuning of Quantized LLMs

arxiv.org

Post reply on HN
Print view View original

61–70 of 116 posts

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#61

Post by hnjst »

pizza wrote: post #26
srslack wrote: post #14

This is off-topic, but are there any communities or congregations (that aren't reddit) based around locally hosted LLMs? I'm asking because while I see a bunch of projects for exposing GGML/LLaMA to OpenAI compatible interfaces, some UIs, etc, I can't really find a good community or resources for the concept in general. I'm working on a front-end for LLMs in general, having re-implemented a working version of OpenAI'…

I imagine you might find your answer in the form of a discord community

https://discord.gg/dwXDHMGY for example.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#62

Post by yyyk »

api wrote: post #8
simonw wrote: post #2

I'm very impressed at the quality of Guanaco 33B, the model that accompanies this paper. You can try it out here: https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi I tried "You are a sentient cheesecake that teaches people SQL, with cheesecake analogies to illustrate different points. Teach me to use count and group by" and got a good result from it: https://twitter.com/simonw/status/1661460336334241794/phot…

Altman’s push for regulatory capture makes so much sense given how fast this field is going. Open models you can run on regular hardware are still behind GPT-4 by some distance but they are closing in at a rate that leads me to believe there’s not much of a moat there. There’s also a ton of promising work on quantization and pruning and other acceleration and compression techniques to make more powerful models run on…

>Open models... are still behind GPT-4 by some distance but they are closing in at a rate that leads me to believe there’s not much of a moat there

Commercially, this doesn't matter even if true. As I keep reminding here: moats are rarely about raw tech. Moats are much more about integrations, brand ("Nobody got fired for buying X"), access to API/raw compute...

If OpenAI have a superior Office 365 integration they'll have a de facto moat. If OpenAI have a larger plugin ecosystem they'll have a de facto moat. If OpenAI has access to better compute (that's far from a certainty) they'll have a moat. If it's much easier to use OpenAI than install a local model they'll have a moat, etc. And that's true even if they don't improve their model at all.

What will open source offer? Privacy? You can see for how little and how easily people barter that.

Moats do fall - but for that, FOSS will have to think product. We're not there at all at the moment.

[EDIT: Oh, I didn't notice the author's name. We've had that conversation in the past here. Sorry for being repetitive.]

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#63

Post by hnjst »

gumballindie wrote: post #28
seydor wrote: post #4

Since Loras are additive, is it possible to use them to do distributed retraining on a model, or even train an entire model bit by bit?

Like a torrent network but for training. That would be cool. The only question is how do you merge changes made by nodes (clients) across the network? Clients could be incentivised to train as they are with crypto, but instead of mining, it's model training and in return they get "coin". Like making crypto mining useful.

https://github.com/bigscience-workshop/petals

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#64

Post by ianpurton »

epups wrote: post #52

Do you know which model size can be run with a 3090?

Assuming you have 24GB of VRAM, then you should be able to run something like 7 billion parameters i.e. MPT-7b and quantized perhaps you can get up to 13 billion.

Note: I have no practical experience of this, just reading around.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#65

Post by scoopertrooper »

imjonse wrote: post #57
ukuina wrote: post #39

Earlier quoted context omitted.

Which is heavier: a pound of feathers, or a great british pound? > Both weights are equal.

So same answer as free ChatGPT, only terser.

GPT-4 hit it out of the park:

The question seems to be asking about two different types of "pounds": one as a unit of weight (the pound of feathers), and one as a unit of currency (the British pound).

A pound of feathers: This is a measure of weight. In the avoirdupois system (which is commonly used in the US), a pound is defined as exactly 0.45359237 kilograms.

A Great British pound: This is the unit of currency in the United Kingdom, often symbolised as £. The weight of a physical £1 coin, according to the Royal Mint, is 8.75 grams.

So, if we are comparing the weight of these two "pounds," a pound of feathers is heavier than a physical £1 coin.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#66

Post by redox99 »

gangwolf wrote: post #50
holtkam2 wrote: post #47

Can someone help me understand what quantization means in this context, and why it matters?

GPT-4 ELI5: - 4-bit Quantization: Imagine you have a box of 16 different colored crayons. But you realize that you can draw almost the same picture using only 4 colors. That's what quantization does. It reduces the number of different "colors" (or numbers) that the model uses to represent its knowledge, which saves a lot of space. In this case, they used a special kind of 4-bit quantization, which means they only use…

It's kind of creepy when GPT4 uses such life-like analogies.

You're an AI, you've never used crayons or played with legos.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#68

Post by [deleted] »

simonw wrote: post #2

I'm very impressed at the quality of Guanaco 33B, the model that accompanies this paper. You can try it out here: https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi I tried "You are a sentient cheesecake that teaches people SQL, with cheesecake analogies to illustrate different points. Teach me to use count and group by" and got a good result from it: https://twitter.com/simonw/status/1661460336334241794/phot…

[deleted]

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#69

Post by ithkuil »

yyyk wrote: post #62
api wrote: post #8

Earlier quoted context omitted.

Altman’s push for regulatory capture makes so much sense given how fast this field is going. Open models you can run on regular hardware are still behind GPT-4 by some distance but they are closing in at a rate that leads me to believe there’s not much of a moat there. There’s also a ton of promising work on quantization and pruning and other acceleration and compression techniques to make more powerful models run on…

>Open models... are still behind GPT-4 by some distance but they are closing in at a rate that leads me to believe there’s not much of a moat there Commercially, this doesn't matter even if true. As I keep reminding here: moats are rarely about raw tech. Moats are much more about integrations, brand ("Nobody got fired for buying X"), access to API/raw compute... If OpenAI have a superior Office 365 integration they'l…

> Open models... are still behind GPT-4 by some distance but they are closing in at a rate that leads me to believe there’s not much of a moat there

This assumes that openai internally doesn't also "closing-in" towards something even more impressive to be released next year or whatnot.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#70

Post by dharma1 »

gangwolf wrote: post #50
holtkam2 wrote: post #47

Can someone help me understand what quantization means in this context, and why it matters?

GPT-4 ELI5: - 4-bit Quantization: Imagine you have a box of 16 different colored crayons. But you realize that you can draw almost the same picture using only 4 colors. That's what quantization does. It reduces the number of different "colors" (or numbers) that the model uses to represent its knowledge, which saves a lot of space. In this case, they used a special kind of 4-bit quantization, which means they only use…

Better than most humans would have come up with! How does this not have some kind of a model of the world, with real world analogies like this?
Post reply on HN