Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

QLoRA: Efficient Finetuning of Quantized LLMs

arxiv.org

Post reply on HN
Print view View original

21–30 of 116 posts

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#21

Post by anon178 »

srslack wrote: post #14

This is off-topic, but are there any communities or congregations (that aren't reddit) based around locally hosted LLMs? I'm asking because while I see a bunch of projects for exposing GGML/LLaMA to OpenAI compatible interfaces, some UIs, etc, I can't really find a good community or resources for the concept in general. I'm working on a front-end for LLMs in general, having re-implemented a working version of OpenAI'…

The /g/ board on 4chan has a /lmg/ general that focuses on running models locally. They regularly discuss fine tuning models, quantization tech, and building apps on text-generation-webui/kobold.

You might get some interest but it's also 4chan...

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#22

Post by thorum »

srslack wrote: post #14

This is off-topic, but are there any communities or congregations (that aren't reddit) based around locally hosted LLMs? I'm asking because while I see a bunch of projects for exposing GGML/LLaMA to OpenAI compatible interfaces, some UIs, etc, I can't really find a good community or resources for the concept in general. I'm working on a front-end for LLMs in general, having re-implemented a working version of OpenAI'…

I'd love to hear more about your approach for getting LLMs to understand how to use plugin commands. My own experiments have not worked very well (even vanilla ChatGPT through the gpt-3.5-turbo API doesn't seem to get the concept, most of the time).

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#23

Post by skybrian »

api wrote: post #17
sva_ wrote: post #11

Earlier quoted context omitted.

To be fair, Altman has been fairly outspoken about not limiting open source models. In how far that was just for streetcred, I cannot say however.

If progress really continues like this any regulation that does not limit open models would be a pointless exercise. It’s only a matter of time before people really crack distributed training algorithms that can be run in a less organized swarm configuration. At that point open trainers could actually train near the frontier of what is possible. Most of the data is open.

Whether or not the regulation goes anywhere, OpenAI keeps doing what they're doing, and they look good to politicians for being cooperative.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#24

Post by sp332 »

orost wrote: post #20
andy_xor_andrew wrote: post #15

Hold on. I need someone to explain something to me. The colab notebook shows an example of loading the vanilla, unquantized model "decapoda-research/llama-7b-hf", using the flag "load_in_4bit" to load it as 4bits. When... when did this become possible? My understanding, from playing with these models daily for the past few months, is that quantization of LLaMA-based models is done via this: https://github.com/qwopqwo…

Quantization isn't (and wasn't) expensive, it's mostly just data shuffling. A good PC will do a 7B model in half a minute, up to a few minutes for a larger model. Quantized models being made available for download is more for the benefit of less technical users who may not be comfortable with the command-line tools, or for people with slow or metered connections who'd much rather download 15GB of data than download 6…

The question is whether this step is actually doing the GPTQ optimized quantization, or simple truncation.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#25

Post by MarcoZavala »

anon178 wrote: post #21
srslack wrote: post #14

This is off-topic, but are there any communities or congregations (that aren't reddit) based around locally hosted LLMs? I'm asking because while I see a bunch of projects for exposing GGML/LLaMA to OpenAI compatible interfaces, some UIs, etc, I can't really find a good community or resources for the concept in general. I'm working on a front-end for LLMs in general, having re-implemented a working version of OpenAI'…

The /g/ board on 4chan has a /lmg/ general that focuses on running models locally. They regularly discuss fine tuning models, quantization tech, and building apps on text-generation-webui/kobold. You might get some interest but it's also 4chan...

[dead]

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#26

Post by pizza »

srslack wrote: post #14

This is off-topic, but are there any communities or congregations (that aren't reddit) based around locally hosted LLMs? I'm asking because while I see a bunch of projects for exposing GGML/LLaMA to OpenAI compatible interfaces, some UIs, etc, I can't really find a good community or resources for the concept in general. I'm working on a front-end for LLMs in general, having re-implemented a working version of OpenAI'…

I imagine you might find your answer in the form of a discord community

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#27

Post by gumballindie »

dharma1 wrote: post #18

fantastic. Will keep my 3090 busy for a while! "Furthermore, we note that our model is only trained with cross-entropy loss (supervised learning) without relying on reinforcement learning from human feedback (RLHF). This calls for further investigations of the tradeoffs of simple cross-entropy loss and RLHF training. " Does this mean RLHF is not really necessary for high quality chatbots?

The RTX3090 is crackin' it. I am considering a dual set-up with nvlink.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#28

Post by gumballindie »

seydor wrote: post #4

Since Loras are additive, is it possible to use them to do distributed retraining on a model, or even train an entire model bit by bit?

Like a torrent network but for training. That would be cool. The only question is how do you merge changes made by nodes (clients) across the network?

Clients could be incentivised to train as they are with crypto, but instead of mining, it's model training and in return they get "coin". Like making crypto mining useful.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#29

Post by gumballindie »

CuriouslyC wrote: post #16
seydor wrote: post #4

Since Loras are additive, is it possible to use them to do distributed retraining on a model, or even train an entire model bit by bit?

Yes, in a sense. It won't work well for distribution of training on a single subject, but you distribute training based on subject and then combine LoRAs.

> and then combine LoRAs

How do you combine them?

> but you distribute training based on subject

Perhaps this could work like a set of hashed and trusted data sets split but subject and topic? Each node downloads one at random and trains against that single subset of a topic or something.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#30

Post by sanxiyn »

andy_xor_andrew wrote: post #15

Hold on. I need someone to explain something to me. The colab notebook shows an example of loading the vanilla, unquantized model "decapoda-research/llama-7b-hf", using the flag "load_in_4bit" to load it as 4bits. When... when did this become possible? My understanding, from playing with these models daily for the past few months, is that quantization of LLaMA-based models is done via this: https://github.com/qwopqwo…

For quality, GPTQ-for-LLaMa repository README is already updated for comparison with this work. See under "GPTQ vs bitsandbytes".
Post reply on HN