Hold on. I need someone to explain something to me. The colab notebook shows an example of loading the vanilla, unquantized model "decapoda-research/llama-7b-hf", using the flag "load_in_4bit" to load it as 4bits. When... when did this become possible? My understanding, from playing with these models daily for the past few months, is that quantization of LLaMA-based models is done via this: https://github.com/qwopqwo…
For quality, GPTQ-for-LLaMa repository README is already updated for comparison with this work. See under "GPTQ vs bitsandbytes".
QLoRA: Efficient Finetuning of Quantized LLMs
41–50 of 116 posts
Re: QLoRA: Efficient Finetuning of Quantized LLMs
#42This is off-topic, but are there any communities or congregations (that aren't reddit) based around locally hosted LLMs? I'm asking because while I see a bunch of projects for exposing GGML/LLaMA to OpenAI compatible interfaces, some UIs, etc, I can't really find a good community or resources for the concept in general. I'm working on a front-end for LLMs in general, having re-implemented a working version of OpenAI'…
I'd love to hear more about your approach for getting LLMs to understand how to use plugin commands. My own experiments have not worked very well (even vanilla ChatGPT through the gpt-3.5-turbo API doesn't seem to get the concept, most of the time).
I'll do a show HN probably at the beginning of next month after this hackathon, but basically look into langchain's "tools" and the different agents they have. You don't need langchain for this at all, but it gives you the groundwork.
I saw their Code Interpreter demo on Twitter (converting an uploaded video file in a chat UI) and decided that I need that, without continuing to pay them money (because they still haven't given me access to it yet.)
So, that, and after sam a went in front of congress for the regulatory capture play, was the motivation I needed to work towards commoditizing these fuckers.
The secret sauce here with code interpreter is, well, literally a python code interpreter you can run in your browser, and it's not so secret.
Re: QLoRA: Efficient Finetuning of Quantized LLMs
#43Earlier quoted context omitted.
If progress really continues like this any regulation that does not limit open models would be a pointless exercise. It’s only a matter of time before people really crack distributed training algorithms that can be run in a less organized swarm configuration. At that point open trainers could actually train near the frontier of what is possible. Most of the data is open.
> It's only a matter of time before people really crack distributed training algorithms that can be run in a less organized swarm configuration. Why do you think this? This continually failed, and seems extremely unlikely to me. Barring surprising breakthrough, there is inherent communication complexity, and physical limit to communication bandwidth.
Yeah the network bandwidth is insane. Each A100 in an 8x A100 (80GB) pod [1] has its own 200 gigabit NIC! Including storage that's nine 200gbps interfaces for almost two terabit of total internal bandwidth.
The newer H100s each have a 400gbit NIC.
[1] https://shop.lambdalabs.com/deep-learning/servers/hyperplane...
Re: QLoRA: Efficient Finetuning of Quantized LLMs
#44Hold on. I need someone to explain something to me. The colab notebook shows an example of loading the vanilla, unquantized model "decapoda-research/llama-7b-hf", using the flag "load_in_4bit" to load it as 4bits. When... when did this become possible? My understanding, from playing with these models daily for the past few months, is that quantization of LLaMA-based models is done via this: https://github.com/qwopqwo…
load_in_4bit requires storing a fully unquantized model as well as having enough RAM to load the unquantized model.
If you're an enthusiast with 10 models downloaded, do you want that taking up 500GB or 150GB? Do you want to need 64GB of RAM to load a model, or just 16GB?
That's the main reason for the popularity of pre-quantization.
Re: QLoRA: Efficient Finetuning of Quantized LLMs
#45This is off-topic, but are there any communities or congregations (that aren't reddit) based around locally hosted LLMs? I'm asking because while I see a bunch of projects for exposing GGML/LLaMA to OpenAI compatible interfaces, some UIs, etc, I can't really find a good community or resources for the concept in general. I'm working on a front-end for LLMs in general, having re-implemented a working version of OpenAI'…
The "lmg" (Local Models General) thread on 4chan's technology board /g/[0] is the premiere community communication spot for open source models, believe it or not.
Everyone from the infamous "oobabooga" to llama.cpp's Georgi Gerganov regularly hangs out in the thread.
If you have questions, you will get answers there.
Re: QLoRA: Efficient Finetuning of Quantized LLMs
#46This is off-topic, but are there any communities or congregations (that aren't reddit) based around locally hosted LLMs? I'm asking because while I see a bunch of projects for exposing GGML/LLaMA to OpenAI compatible interfaces, some UIs, etc, I can't really find a good community or resources for the concept in general. I'm working on a front-end for LLMs in general, having re-implemented a working version of OpenAI'…
The "lmg" (Local Models General) thread on 4chan's technology board /g/[0] is the premiere community communication spot for open source models, believe it or not. Everyone from the infamous "oobabooga" to llama.cpp's Georgi Gerganov regularly hangs out in the thread. If you have questions, you will get answers there. [0] https://boards.4channel.org/g/#s=lmg
You know HN has gotten lackluster when 4chan is more informed.
Re: QLoRA: Efficient Finetuning of Quantized LLMs
#47Can someone help me understand what quantization means in this context, and why it matters?
Re: QLoRA: Efficient Finetuning of Quantized LLMs
#48Can someone help me understand what quantization means in this context, and why it matters?
Quantizing is porting the weights from using high-precision floating point decimals to lower-precision numbers, e.g., 4-bit and 8-bit ints. Less memory used by the weights means we can run bigger models on the same hardware.
The idea behind quantization is that these models have so many parameters, they'll still work even if we reduce each node's flexibility.
Re: QLoRA: Efficient Finetuning of Quantized LLMs
#49Earlier quoted context omitted.
To be fair, Altman has been fairly outspoken about not limiting open source models. In how far that was just for streetcred, I cannot say however.
If progress really continues like this any regulation that does not limit open models would be a pointless exercise. It’s only a matter of time before people really crack distributed training algorithms that can be run in a less organized swarm configuration. At that point open trainers could actually train near the frontier of what is possible. Most of the data is open.
The wording of this would be extremely difficult though. Are local NER models part of this? Relation extraction? What about GPTs that only decode to DSLs? If the model only outputs DNA sequences is that an area that can be more illegal or less if done for research by an individual?
The breadth of different tasks and architectures can make this exceedingly challenging to regulate.
Re: QLoRA: Efficient Finetuning of Quantized LLMs
#50Can someone help me understand what quantization means in this context, and why it matters?
GPT-4 ELI5:
- 4-bit Quantization: Imagine you have a box of 16 different colored crayons. But you realize that you can draw almost the same picture using only 4 colors. That's what quantization does. It reduces the number of different "colors" (or numbers) that the model uses to represent its knowledge, which saves a lot of space. In this case, they used a special kind of 4-bit quantization, which means they only used 16 different numbers instead of the thousands or millions that the model might usually use.
- Low Rank Adapters (LoRA): This is a way to change the model's knowledge without having to touch every piece of it. Imagine you have a huge, complicated Lego structure, and you want to change it. Instead of taking apart the whole thing, you just add or change a few pieces here and there. That's what LoRA does. It allows the researchers to fine-tune the model without having to use as much memory.
- Double Quantization: This is another trick to save memory. It's like if you realized that you could represent each of your 4 crayon colors with just 2 symbols, so you save even more space.
- Paged Optimizers: This is a way to handle moments when the model needs a lot of memory all at once. It's like if you have a small desk, but sometimes you need to work on a big project. Instead of getting a bigger desk, you just clear off and use the desk in small sections at a time.
By using these techniques, the researchers were able to train a very large model (Guanaco) on a single graphics card, which would normally not have enough memory for this task.