just ask it what is qlora and it will give wrong answers
QLoRA: Efficient Finetuning of Quantized LLMs
111–116 of 116 posts
Re: QLoRA: Efficient Finetuning of Quantized LLMs
#112Hold on. I need someone to explain something to me. The colab notebook shows an example of loading the vanilla, unquantized model "decapoda-research/llama-7b-hf", using the flag "load_in_4bit" to load it as 4bits. When... when did this become possible? My understanding, from playing with these models daily for the past few months, is that quantization of LLaMA-based models is done via this: https://github.com/qwopqwo…
Very, very recently. In the past few days. I tried it out immediately because GPTQ-for-LLaMA and hunting for or making quantized models can be tedious, but it was disappointingly slow. On a 3090 where I was getting responses for a given 13B model in 10-30 seconds, just using transformers with load_in_4bit took about ten times that for each response.
There’s also the storage benefit of using actually quantized models.
Re: QLoRA: Efficient Finetuning of Quantized LLMs
#113Earlier quoted context omitted.
Like a torrent network but for training. That would be cool. The only question is how do you merge changes made by nodes (clients) across the network? Clients could be incentivised to train as they are with crypto, but instead of mining, it's model training and in return they get "coin". Like making crypto mining useful.
https://github.com/bigscience-workshop/petals
Saw that - a real cool project! Thanks for sharing! Now the question is how to motivate people to train models?
I am thinking of somehow replacing crypto currency hashing with model training and inference. Somehow a training topic and data get fed into the network - naturally it somehow must be filtered to avoid weird crap - and queries are somehow run. Imagine all the crypto mining farms and gpus switching over to training massive models.
Re: QLoRA: Efficient Finetuning of Quantized LLMs
#114Earlier quoted context omitted.
I alluded to what you say in the last paragraph: the average user isn't going to play with open source models, but the existence of the open source AI ecosystem will enable more competitors. There's not going to just be one AI company. There will be thousands and thousands, each addressing different use cases and market niches. In a world where OpenAI has a powerful technological moat, all of these companies would en…
I agree OpenAI won't and can't have anywhere near a monopoly, excluding very unlikely legal shenanigans. The field is too approachable, and ultimately, to pinch into their market one doesn't need GPT-4 level performance. Besides, apparently OpenAI can't scale compute fast enough, so users will end up by default requiring other providers. There's a very good chance of OpenAI/Microsoft being major players though IMHO.…
Interesting parallels , there is good possibility for this outcome with respect to Open Source LLMs
> I can however see a possible future where open source is not going to have any significant impact on LLMs, say like Desktop Linux.
> Either because it gets stuck in a technical realm and doesn't make anything too approachable to ordinary users, or because it lacks the necessary integrations..
Re: QLoRA: Efficient Finetuning of Quantized LLMs
#115Earlier quoted context omitted.
RAM, not VRAM. The model is fully loaded into (regular, system) RAM then quantized while being loaded into VRAM. So you still need enough system RAM (or RAM+Swap) to load the unquantized model.
Why would it need to be loaded fully into system RAM at once? Couldn't it be quantized on the fly, as it's being read from disk?
Last time I looked into this, the answer was "because huggingface transformers and torch.load are written to do it this way"
You could absolutely do something streaming, or mmap the weights instead of loading them into system RAM. Just the default interfaces don't.
Re: QLoRA: Efficient Finetuning of Quantized LLMs
#116Earlier quoted context omitted.
I agree that pure open source probably won't cross over into general use. Open source is great at tech but bad at product, mostly because product requires doing a lot of boring polish that programmers hate and thus generally have to be paid to do (I have ranted about this extensively here and elsewhere). Consumers also hate choice these days because they're all in a state of continuous cognitive overload. They want t…
Well, I guess we agree here - the basic functionality will be common, Open AI will not have a monopoly (unless lawmakers really screw up), but will likely exist and be a major player. I am not sure about Open Source at all just yet - e.g. it's possible that a panic will lead to restrictions that will make open work rather difficult. I guess we'll see. >what happens when everyone uses AI? One of the most exciting poss…
Thinking about this a bit more, the main issue is that the LLM is an attack vector. Hallucinations aren't a big deal if the action requires approval or can be easily undone. The only way right now I can see to make LLM interfaces 'safe' would be to give it an internal API and pass the user context so we can set up permissions that way. There are a very few cases where even that might not be enough (basically whenever you pass 'on-behalf-of' transactions _without_ direct approval from the origin via a temporary token - e.g. many backend payment APIs allow this).
However, if you have an internal API which is the real security boundary, you might as well expose it. The effort required to add an LLM is now an extra effort and that flips the incentives to where the LLM really adds value. So adding LLMs as interfaces really makes sense on Windows 11 (practically single user, hallucinations have limited cost) and GitHub (devs have local copy anyway), maybe on AWS (devs would like an easier interface, but there's a need to reassure about safety), and none on Stripe or MailGun (complex security scenarios, usually decided by finance/marketing department which don't care about integration difficulty).