Does quantizing the models reduce their "accuracy"?
Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
221–230 of 298 posts
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#222How long until someone puts out a docker image of this?
Ideally with a nice rest API, I can't imagine it's too hard to do.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#223Earlier quoted context omitted.
>20 tokens per second (~4 words per second) How can there be 5 tokens per word, when they have more than half the vocabulary as GPT-2/3 which has 1.3 tokens per word? I would have guessed more like 1.5 tokens per word.
Oh, it’s probably higher than four words per second, then. I assumed tokens was characters and used the standard “there are five characters in a word” rule of thumb.
It's about 4 charcters per token. So just over 1 token per word. I just round to 1 token per word since text most people generate does not use larger words and because larger common words are still encoded as one token (e.g. HackerNews is probably one token despite being 10 characters).
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#224Earlier quoted context omitted.
LLaMA it doesn't require any system RAM to run. It requires some very minimal system RAM to load the model into VRAM and to compile the 4bit quantized weights. But if you use pre-quantized weights (get them from HuggingFace or a friend) then all you really need is ~32GB of VRAM and maybe around 2GB of system RAM for 65B. (It's 30B which needs 20GB of VRAM.)
The full use case includes quantisation, which the repo points out uses a large amount of system RAM. Of course that’s not required if you skip that step.
Judging from downloads of the 4bit file and how many people I've seen post about quantizing it themselves, around 99% of people are just downloading the pre-quantized files.
I would not personally call compilation of software part of its "use case." It's use case is text generation.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#225Earlier quoted context omitted.
Won't the 65b model (almost) fit into 128GB RAM? Or into 128GB RAM and 24GB VRAM?
Yes (I just don't have that much ram) I have a separate branch that streams weights from ram - at which point I think I was only seeing negligible performance loss compared to storing the weights in vram. The bottleneck was compute, not GPU bandwidth.
The 65B model only needs just over 32GB of VRAM to run. It does not need system RAM to run/use if you use pre-quantized weights which you can find many places already.
No need to quantize yourself (besides it takes almost a day to do 4bit GPTQ quantization on 3xA6000).
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#226how long before someone creates a simple GUI for this? That + a small bit of optimisation and everyone with a newer Mac / iPhone will be able to run something akin to chatGPT locally! Isn't this a pretty crazy development - just weeks ago people said this would be impossible. From this thread the 13b model runs just as fast as chatGPT on a M2 Macbook Air, and it's not even using the Neural Engine yet so will become s…
People have been running LLaMA in 4bit quickly on cheap hardware with a simple GUI for over a week using https://github.com/oobabooga/text-generation-webui
Just not on Macs. (that repo does not support Apple Silicon)
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#227Earlier quoted context omitted.
You're half right. First, tensor operations are only a small part of modern ML. Second, how you plug all those small operation together is where all the performance difference is had these days between implementations. Different hardware have a variety of different small operations that do almost the same thing. So when a state of the art model architecture meets a state of the art quantization method and you want to…
Ok, but in the end you're just evaluating a graph, and I suppose that compilers can figure out how to do this in the most efficient way on any type of hardware for which a backend was written. So it makes more sense to work on a backend that you can use for any type of model than to hand-optimize everything.
>I suppose that compilers can figure out how to do this in the most efficient way on any type of hardware for which a backend was written.
No, that's exactly the problem. Compilers can't because the GPU hardware and the algorithms involved are such rapidly moving targets. Bespoke hardware specific quantization, inference, attention, and kernel compilation is the only way to squeeze out the performance users are looking for.
Creating one fast implementation for all models on all hardware would be like writing one GPU driver for all GPUs and OSs. It just isn't going to work and if it does it isn't going to be fast on all hardware.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#228Does quantizing the models reduce their "accuracy"?
Yes, but only minimally. Not enough for any human to notice.
However, even this minimal amount can be avoided with GPTQ quantization which maintains uncompressed fp16 performance even at 4bit quantization with 75% less (video)memory overhead.
References:
https://arxiv.org/abs/2210.17323 - GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers [Oct, 2022]
https://arxiv.org/abs/2212.09720 - The case for 4-bit precision: k-bit Inference Scaling Laws [Dec, 2022]
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#229Does anyone know how many languages this supports? I know that FB has been translated to a ton of languages. Will those translations benefit the models in LLaMA? Or am I misunderstanding the point of this?
The point of it is to generate arbitrary text locally, like the GPT-3 API does remotely.
It can "say"/talk about anything an average IQ person with knowledge of the entire internet and most books in existence could. So if you prompt it to write 100 pages on the differences between positive and negative law, as a poem, while never using a word with the letter "F" it will spit that out for you without any issue.
It can also program quite well, create recipes, debate with you, impersonate anyone, and lots more. And it does all of this offline, in airplane mode, locally on your PC or Mac.
It's good at translation but is probably one of the least efficient ways to translate text when models specifically for translation exist.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#230> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing. With a lot of fine tuning if y…
Wonder why AMD/Intel/Nvidia haven’t invented some sort of device that allows the processor and graphics to share memory like Apple has done.
Games don't want slow memory accesses because it tanks the fps so there was no incentive to have a working implementation (except in laptops).
And also - vram is a moat that keeps the cost of "professional" cards absurdly high without making them actually faster than consumer cards.