Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Quantized Llama models with increased speed and a reduced memory footprint

ai.meta.com

Post reply on HN
Print view View original

81–90 of 128 posts

Re: Quantized Llama models with increased speed and a reduced memory footprint

#81

Post by imjonse »

formalsystem wrote: post #47
philipkglass wrote: post #45

Earlier quoted context omitted.

What was the "vanilla post-training quantization" used for comparison? There are 22 GGUF quantization variants smaller than 16 bits per weight and I can't tell which one is being compared with: https://huggingface.co/docs/hub/en/gguf#quantization-types It might even mean a non-GGUF quantization scheme; I'm just an intermediate user of local models, not an expert user or developer.

So this should be referring to w8a8 (weights and activations in 8 bit) So this is gonna be 8 bit weights, 8 bit activations, group size of 256, symmetric quantization. Not sure how to map this to the GGUF variants because they don't mention how they don't do activation quantization

Were there comparisons made to AWS, Smoothquant, GPTQ or other non-vanilla PTQ methods? Thanks.

Re: Quantized Llama models with increased speed and a reduced memory footprint

#82

Post by formalsystem »

imjonse wrote: post #81
formalsystem wrote: post #47

Earlier quoted context omitted.

So this should be referring to w8a8 (weights and activations in 8 bit) So this is gonna be 8 bit weights, 8 bit activations, group size of 256, symmetric quantization. Not sure how to map this to the GGUF variants because they don't mention how they don't do activation quantization

Were there comparisons made to AWS, Smoothquant, GPTQ or other non-vanilla PTQ methods? Thanks.

Not that I know of for this study, at least for the specific scope torchao we want to make it easier for researchers to create new quantization algorithms in python and have those algorithms run fast and you can see a lot of those algorithms here https://github.com/pytorch/ao/tree/main/torchao/prototype

So for example for AWQ and GPTQ we can accelerate them by using a fast int4 kernel called tinygemm

Re: Quantized Llama models with increased speed and a reduced memory footprint

#83

Post by formalsystem »

Evidlo wrote: post #75
formalsystem wrote: post #43

Hi I'm Mark I work on torchao which was used for the quantization aware training and ARM kernels in this blog. If you have any questions about quantization or performance more generally feel free to let me know!

I have a non-ML question. In vanilla Pytorch I have the following expression: t.sum(values[inds] * weights) If 'inds' is int8, I get "IndexError: tensors used as indices must be long, int, byte or bool tensors". Is this still true if I use torchao?

The issue here is memory in PyTorch is byte addressable and that's a limitation we can't solve without making a lot more changes to PyTorch. But in your specific case, if you'd like to pack more data into `values` you can use a combination of clever bit shifting, torch.cat and other bit twiddling pytorch like ops to pack more data. It's a trick we use quite heavily in torchao

Re: Quantized Llama models with increased speed and a reduced memory footprint

#85

Post by theanonymousone »

ipsum2 wrote: post #80
theanonymousone wrote: post #9

May I ask if anyone has successfully used 1B and 3B models in production and if yes, in what use cases? I seem to be failing even in seemingly simpler tasks such as word translation or zero-shot classification. For example, they seem to not care about instructions to only write a response and no explanation, thus making it impossible to use them in a pipeline :/

You can't expect a 1B model to perform as well as 7B or chatGPT, probably the best use case is speculative decoding or to use to fine tune for a specific use case.

What is "speculative decoding"?

Re: Quantized Llama models with increased speed and a reduced memory footprint

#86

Post by 127 »

derefr wrote: post #58
tveita wrote: post #37

So SpinQuant learns a rotation for activations and weights that, to my understanding, "smear" the outlier weights out so you don't get extreme values in any one weight. Random anecdote warning - In the old days, before vector search became AI and everyone and their dog offered a vector database, I had a task that required nearest neighbour search in a decent amount of high-dimensional vectors. I tried quantizing them…

> But it's a pretty rare day at work that "apply a random rotation matrix to a 128-dimensional vector" is the solution to my problem. Funny enough, if you visualize a vector-embedding's latent-space features using that "points on the surface of a hypersphere" analogy that ML programmers like to use — and you assume a really low quantization, say, 1-bit — then you can almost picture the hypersphere surface as a black-…

The best type of dithering is done with error diffusion. There's a convolutional kernel the diffuses the error over multiple adjacent data points.

Re: Quantized Llama models with increased speed and a reduced memory footprint

#87

Post by pizza »

slimsag wrote: post #56
o11c wrote: post #34

Earlier quoted context omitted.

Because the way LLMs work is more-or-less "for every token, read the entire matrix from memory and do math on it". Math is fast, so if you manage to use only half the bits to store each item in the matrix, you only have to do half as much work. Of course, sometimes those least-significant-bits were relied-upon in the original training.

Has anyone worked on making tokens 'clusters of words with specific semantic meaning'? e.g. instead of tokens ['i', 'am', 'beautiful'] having tokens ['I am', 'beautiful'] on the premise that 'I am' is a common set of bytes for a semantic token that identifies a 'property of self'? Or taking that further and having much larger tokens based on statistical analysis of common phrases of ~5 words or such?

I think you might be thinking of applying a kind of low-rank decomposition to the vocabulary embeddings. A quick search on Google Scholar suggests that this might be useful in the context of multilingual tokenization.

Re: Quantized Llama models with increased speed and a reduced memory footprint

#88

Post by regularfry »

eru wrote: post #67
eirikbakke wrote: post #65

Earlier quoted context omitted.

Fascinating! Does that mean you could improve performance further with Floyd–Steinberg dithering? (I.e. instead of rotating randomly, you track accumulated quantization error and add that amount instead.)

Floyd-Steinberg etc mostly look better to the human eye, but I'm not sure in what more 'objective' sense they would be better than random dithering?

Floyd-Steinberg is one sort of quasi-random algorithm, but there are others. People often use quasi-random rather than true randomness when they want to avoid sample points bunching together. They tend to be more evenly distributed. That can get more important in higher-dimension space where it's easy to completely miss sampling large volumes because a truly random point set has too many degrees of freedom.

Re: Quantized Llama models with increased speed and a reduced memory footprint

#89

Post by regularfry »

theanonymousone wrote: post #85
ipsum2 wrote: post #80

Earlier quoted context omitted.

You can't expect a 1B model to perform as well as 7B or chatGPT, probably the best use case is speculative decoding or to use to fine tune for a specific use case.

What is "speculative decoding"?

Speculative decoding is using a small model to quickly generate a sequence that every so often you pass through a larger model to check and correct. It can be much faster than just using the larger model, with tolerably close accuracy.

Re: Quantized Llama models with increased speed and a reduced memory footprint

#90

Post by regularfry »

behnamoh wrote: post #23

Does anyone know why the most common method to speed up inference time is quantization? I keep hearing about all sorts of new methods but nearly none of them is implemented in practice (except for flash attention).

In addition to the other answers in this thread, there's a practical one: sometimes (ok, often) you want to run a model on a card that doesn't have enough VRAM for it. Quantisation is a way to squeeze it down so it fits. For instance I've got a 4090 that won't fit the original Llama3 70b at 16 bits per param, but it will give me usable token rates at 2 bits.
Post reply on HN