[flagged]
[deleted]
Quantized Llama models with increased speed and a reduced memory footprint
71–80 of 128 posts
Re: Quantized Llama models with increased speed and a reduced memory footprint
#72Hi I'm Mark I work on torchao which was used for the quantization aware training and ARM kernels in this blog. If you have any questions about quantization or performance more generally feel free to let me know!
Do you ever pronounce torchao in a way that rhymes with "wow"
Re: Quantized Llama models with increased speed and a reduced memory footprint
#73Earlier quoted context omitted.
Because the way LLMs work is more-or-less "for every token, read the entire matrix from memory and do math on it". Math is fast, so if you manage to use only half the bits to store each item in the matrix, you only have to do half as much work. Of course, sometimes those least-significant-bits were relied-upon in the original training.
Has anyone worked on making tokens 'clusters of words with specific semantic meaning'? e.g. instead of tokens ['i', 'am', 'beautiful'] having tokens ['I am', 'beautiful'] on the premise that 'I am' is a common set of bytes for a semantic token that identifies a 'property of self'? Or taking that further and having much larger tokens based on statistical analysis of common phrases of ~5 words or such?
yes, look up Byte Pair Encoding
Re: Quantized Llama models with increased speed and a reduced memory footprint
#74Hi I'm Mark I work on torchao which was used for the quantization aware training and ARM kernels in this blog. If you have any questions about quantization or performance more generally feel free to let me know!
Do you ever pronounce torchao in a way that rhymes with "wow"
My wife calls it torch AAAW
Re: Quantized Llama models with increased speed and a reduced memory footprint
#75Hi I'm Mark I work on torchao which was used for the quantization aware training and ARM kernels in this blog. If you have any questions about quantization or performance more generally feel free to let me know!
I have a non-ML question.
In vanilla Pytorch I have the following expression:
t.sum(values[inds] * weights)
Is this still true if I use torchao?
Re: Quantized Llama models with increased speed and a reduced memory footprint
#76Why don't they actually say what the size of the model is in GB?
That and average inference times on common hardware is what I'm curious about.
Re: Quantized Llama models with increased speed and a reduced memory footprint
#77It's pretty interesting that the new SpinQuant method did not manage to be better than good old nf4bit QLORA training (Tim Dettmers really cooked with that one). Really appreciate that Meta published both results+model quants and didn't just make some bs claim about a new sota quant like most other bigger companies would've done.
The naming is unfortunate but in this blog QLoRA is referring to Quantization-Aware Training with LoRA adaptor
Re: Quantized Llama models with increased speed and a reduced memory footprint
#78Why don't they actually say what the size of the model is in GB? That and average inference times on common hardware is what I'm curious about.
The last table shows memory usage and performance on an Android phone.
> Decode latency improved by 2.5x and prefill latency improved by 4.2x on average, while model size decreased by 56% and memory usage reduced by 41% on average. The benchmarks can be reproducible today via ExecuTorch Llama instructions. The table above shows results using an Android OnePlus 12 device—however, we’ve also verified similar relative performance on Samsung S24+ for 1B and 3B and Samsung S22 for 1B.
Re: Quantized Llama models with increased speed and a reduced memory footprint
#79It's pretty interesting that the new SpinQuant method did not manage to be better than good old nf4bit QLORA training (Tim Dettmers really cooked with that one). Really appreciate that Meta published both results+model quants and didn't just make some bs claim about a new sota quant like most other bigger companies would've done.
Those are different approaches afaict.
Re: Quantized Llama models with increased speed and a reduced memory footprint
#80May I ask if anyone has successfully used 1B and 3B models in production and if yes, in what use cases? I seem to be failing even in seemingly simpler tasks such as word translation or zero-shot classification. For example, they seem to not care about instructions to only write a response and no explanation, thus making it impossible to use them in a pipeline :/
You can't expect a 1B model to perform as well as 7B or chatGPT, probably the best use case is speculative decoding or to use to fine tune for a specific use case.