Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

QLoRA: Efficient Finetuning of Quantized LLMs

arxiv.org

Post reply on HN
Print view View original

91–100 of 116 posts

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#91

Post by newshorts »

redox99 wrote: post #66
gangwolf wrote: post #50

Earlier quoted context omitted.

GPT-4 ELI5: - 4-bit Quantization: Imagine you have a box of 16 different colored crayons. But you realize that you can draw almost the same picture using only 4 colors. That's what quantization does. It reduces the number of different "colors" (or numbers) that the model uses to represent its knowledge, which saves a lot of space. In this case, they used a special kind of 4-bit quantization, which means they only use…

It's kind of creepy when GPT4 uses such life-like analogies. You're an AI, you've never used crayons or played with legos.

I believe LoRA developed after GPT-4’s cutoff date. So this appears to have come from a human…

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#92

Post by zirgs »

yyyk wrote: post #80
zirgs wrote: post #73

Earlier quoted context omitted.

Open models can talk about "forbidden" topics and can be extended by users. Both of them are significant advantages.

The ability to talk about 'forbidden' topics is also a significant dis advantage. Just wait for the first moral panic targeting open source GPTs. I think that Open Source will exist (barring a legal ban), but the triumphalism is in my mind very unjustified. There's a fair chance closed source will get 99% of this market.

It's an advantage, because each user can apply different restrictions to Open Source AI models. If I'm writing a crime novel for adults - I want the assistant to be able to talk about sex and violence. But obviously - a corporate customer support chatbot should not talk about those things.

I understand why ClosedAI added those restrictions. But they are too inflexible.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#93

Post by fnands »

Lol, some residual answers from what I assume is distilled answers from ChatGPT:

Q: "What is your favourite conspiracy theory?" A: "As an AI language model I don’t have personal preferences or biases so my responses will always reflect factual information based on what has been programmed into me by OpenAI."

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#94

Post by seydor »

gumballindie wrote: post #28
seydor wrote: post #4

Since Loras are additive, is it possible to use them to do distributed retraining on a model, or even train an entire model bit by bit?

Like a torrent network but for training. That would be cool. The only question is how do you merge changes made by nodes (clients) across the network? Clients could be incentivised to train as they are with crypto, but instead of mining, it's model training and in return they get "coin". Like making crypto mining useful.

While i understand the motives, they should just train models bits for free, a-la SETI@home. Then let people pick and choose which collections to use to their chatbot. The crypto aspect only adds complexity and wrong incentives that would hold it back. After all it's not possible to verify that each node truly trained the network and is not returning garbage (multiple trainings can be involved, which would make the whole thing slower).

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#95

Post by MacsHeadroom »

amilios wrote: post #84
MacsHeadroom wrote: post #44

Earlier quoted context omitted.

load_in_4bit requires storing a fully unquantized model as well as having enough RAM to load the unquantized model. If you're an enthusiast with 10 models downloaded, do you want that taking up 500GB or 150GB? Do you want to need 64GB of RAM to load a model, or just 16GB? That's the main reason for the popularity of pre-quantization.

Does it? I loaded Guanaco-65B on a 48GB graphics card using the new flag and it loaded just fine. I don't think the unquantized model fits on a 48GB card. So I'm not quite sure if this is true.

RAM, not VRAM. The model is fully loaded into (regular, system) RAM then quantized while being loaded into VRAM.

So you still need enough system RAM (or RAM+Swap) to load the unquantized model.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#97

Post by seydor »

gumballindie wrote: post #89
CuriouslyC wrote: post #40

Earlier quoted context omitted.

We already have exactly that for stable diffusion with Civitai.com. People have published a variety of LoRAs for different subjects just as you describe. The local LLM community is very much following the lead of the stable diffusion community in terms of how it's organizing, so I expect that we'll see a proliferation of domain LoRAs being published on an aggregator for LLM stuff before too long.

I don't think the two concepts are similar. I see no incentive for people to train for civitai and find no particular use for the generated content. Edit: actually, some of that content looks suspicious.

Porn is a major reason that people train loras. The beauty of them is that people can pick and choose multiple loras to build truly custom ai

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#98

Post by janekm »

dsrtslnd23 wrote: post #96

From the paper it looks like you would get GPT-4 level quality with the 65B model - but if you just do some random tests you will quickly figure out that is not even remotely the case. There must be something seriously wrong with the benchmarks used.

The evaluation prompts I saw ( https://colab.research.google.com/drive/1kK6xasHiav9nhiRUJjP... ) were all "encyclopedia-style", asking for summaries of information that would have occurred many times in the training data (e.g. "How can I improve my time management skills?") and hence probably reflect ability to retrieve and summarise trained information as opposed to novel problem solving.

Or in the programming questions the ones I saw are also certainly in the training data (e.g. "Implement a Python function to find the longest common subsequence of two input strings using dynamic programming.")

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#99

Post by heliophobicdude »

dharma1 wrote: post #18

fantastic. Will keep my 3090 busy for a while! "Furthermore, we note that our model is only trained with cross-entropy loss (supervised learning) without relying on reinforcement learning from human feedback (RLHF). This calls for further investigations of the tradeoffs of simple cross-entropy loss and RLHF training. " Does this mean RLHF is not really necessary for high quality chatbots?

Yes and no.

Someone pointed out that some of the answers are using OpenAI's famous "As an AI...". Soooo you can roughly say that RLHF might still have had an impact here through the training data that came from an RLHF model.

But what we are seeing is a revolution in picking quality demonstration data that might make RLHF an optional last step for fine-tuned models.

I'm bullish on a new technique where the quality of instruction-tuned code model output are measured and evaluated automatically by a machine. I'm calling it RLF

Code Model outputs a test harness, an implementation, and how to run the implementation for a users inputs. Have RLMF evaluate implementations against the synthetic test harness and against hidden user inputs.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#100

Post by opyate »

potatoman22 wrote: post #48
holtkam2 wrote: post #47

Can someone help me understand what quantization means in this context, and why it matters?

Quantizing is porting the weights from using high-precision floating point decimals to lower-precision numbers, e.g., 4-bit and 8-bit ints. Less memory used by the weights means we can run bigger models on the same hardware. The idea behind quantization is that these models have so many parameters, they'll still work even if we reduce each node's flexibility.

In the same breath, there's no free lunch. There's always a trade-off. Sure, the model might now fit in your VRAM, but it might be less accurate for your specific task.
Post reply on HN