Quantized Llama models with increased speed and a reduced memory footprint
1–10 of 128 posts
Re: Quantized Llama models with increased speed and a reduced memory footprint
#2TLDR: Quantized versions of Llama 3.2 1B and 3B models with "competitive accuracy" to the original versions (meaning some degraded performance; plots included in the release notes).
Re: Quantized Llama models with increased speed and a reduced memory footprint
#3TLDR: Quantized versions of Llama 3.2 1B and 3B models with "competitive accuracy" to the original versions (meaning some degraded performance; plots included in the release notes).
Quantization schemes include post-training quantization (PTQ), SpinQuant, and QLoRA.
Re: Quantized Llama models with increased speed and a reduced memory footprint
#4How do they compare to their original quants on ollama like q4_K_S?
Re: Quantized Llama models with increased speed and a reduced memory footprint
#5These quantized models show much less degradation compared to a "vanilla post-training-quantization" but there are a bunch of PTQ schemes that people have already applied to Llama models [1]. I didn't see any details about the vanilla PTQ they used as a baseline. Has it been written about elsewhere?
Re: Quantized Llama models with increased speed and a reduced memory footprint
#6How do they compare to their original quants on ollama like q4_K_S?
These undergo additional fine tuning (QLoRA) using some or all of the original dataset, so they're able to get the weights to align to the nf4 dtype better, which increases the accuracy.
Re: Quantized Llama models with increased speed and a reduced memory footprint
#7It's pretty interesting that the new SpinQuant method did not manage to be better than good old nf4bit QLORA training (Tim Dettmers really cooked with that one).
Really appreciate that Meta published both results+model quants and didn't just make some bs claim about a new sota quant like most other bigger companies would've done.
Re: Quantized Llama models with increased speed and a reduced memory footprint
#8Anyone know a nice iOS app to run these locally?
Re: Quantized Llama models with increased speed and a reduced memory footprint
#9May I ask if anyone has successfully used 1B and 3B models in production and if yes, in what use cases? I seem to be failing even in seemingly simpler tasks such as word translation or zero-shot classification. For example, they seem to not care about instructions to only write a response and no explanation, thus making it impossible to use them in a pipeline :/
Re: Quantized Llama models with increased speed and a reduced memory footprint
#10[dead]