Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Quantized Llama models with increased speed and a reduced memory footprint

ai.meta.com

Post reply on HN
Print view View original

121–128 of 128 posts

Re: Quantized Llama models with increased speed and a reduced memory footprint

#121

Post by formalsystem »

philipkglass wrote: post #45
formalsystem wrote: post #43

Hi I'm Mark I work on torchao which was used for the quantization aware training and ARM kernels in this blog. If you have any questions about quantization or performance more generally feel free to let me know!

What was the "vanilla post-training quantization" used for comparison? There are 22 GGUF quantization variants smaller than 16 bits per weight and I can't tell which one is being compared with: https://huggingface.co/docs/hub/en/gguf#quantization-types It might even mean a non-GGUF quantization scheme; I'm just an intermediate user of local models, not an expert user or developer.

Please ignore my previous comments - I double checked with the model developers and here's the correction. Vanilla PTQ means no fancy quantization algorithm like SpinQuant, AWQ, etc. was applied. It just applied the same quantization scheme mentioned in the post (4bit per-group with g_size=32 symmetric weight, 8bit dynamic per token activation).

Re: Quantized Llama models with increased speed and a reduced memory footprint

#122

Post by nikolayasdf123 »

SoLoMo123 wrote: post #116
nikolayasdf123 wrote: post #61

what's your opinion on LlamaStack? for me it is nothing short of bad experience. it is way over-engineered with poor quality and just plain does not work, and maintainers are questionable. I would rather call HuggingFace python code for inference or anything else. is ExecuTorch any better?

Hi, I'm Mergen and I work on ExecuTorch. ExecuTorch is a runtime for mobile and embedded devices to run PyTorch models directly. Currently it runs pretty fast on CPU, but expanding our use-case for mobile accelerators and GPUs. We're still in our early stages (just turned beta status). But try it out and let us know. Regarding Llama Stack, it is built by my colleagues. What were some concrete issues have you experien…

will give executorch a try.

with llamastack, well making it work with CUDA for starters would be great.

it is also bloated. something that supposed to take direct 100 lines of code and a couple files, takes dozens of files, multiple frameworks, generators.. which in the end do not work at all, and nobody knows why. very obscure framework. can't believe this code is coming from Meta.

Re: Quantized Llama models with increased speed and a reduced memory footprint

#123

Post by kgc »

qeternity wrote: post #97
regularfry wrote: post #89

Earlier quoted context omitted.

Speculative decoding is using a small model to quickly generate a sequence that every so often you pass through a larger model to check and correct. It can be much faster than just using the larger model, with tolerably close accuracy.

> with tolerably close accuracy. No, speculative decoding has exactly the same accuracy as the target model. It is mathematically identical to greedy decoding.

Is there a reference for this? I was wondering the same thing.

Re: Quantized Llama models with increased speed and a reduced memory footprint

#124

Post by qeternity »

kgc wrote: post #123
qeternity wrote: post #97

Earlier quoted context omitted.

> with tolerably close accuracy. No, speculative decoding has exactly the same accuracy as the target model. It is mathematically identical to greedy decoding.

Is there a reference for this? I was wondering the same thing.

Read the original whitepaper or go look at how any framework implements it.

You will see that tokens not predicted by greedy sampling of the target model are rejected. Ergo, they are mathematically identical.

Re: Quantized Llama models with increased speed and a reduced memory footprint

#125

Post by ed »

tucnak wrote: post #102
ed wrote: post #42

Oh cool! I’ve been playing with quantized llama 3B for the last week. (4-bit spinquant). The code for spinquant has been public for a bit. It’s pretty adept at most natural language tasks (“summarize this”) and performance on iPhone is usable. It’s even decent at tool once you get the chat template right. But it struggles with json and html syntax (correctly escaping characters), and isn’t great at planning, which ma…

>But it struggles with json You should customise your sampler to mandate JSON grammar after ```json tokens.

Grammar samplers are clever! But in the case of a missing escape character you’ll end up with a corrupted string.

Take for example: "A dog says \"Woof!\""

With a grammar, you’ll end up with "A dog says " when the model forgets to escape.

Which is valid JSON, but not what the model intended.

So it’s usually better to catch the exception and ask the model to try again.

Unless you’ve come across a sampler with backtracking? That would be cool

Re: Quantized Llama models with increased speed and a reduced memory footprint

#126

Post by niutech »

yuvalr1 wrote: post #109

Looking at how to deploy 1B and 3B Llama models on Android for inference. Some posts online recommend using Termux (an amazing app) to have an emulated shell and then install as if it's Linux, using ollama for example. However, this forces you into a manual installation process, and also most of the people don't know what Termux is, and would be afraid to install it from F-Droid. Maybe someone can recommend a way to…

You can use MLC LLM: https://llm.mlc.ai/

Re: Quantized Llama models with increased speed and a reduced memory footprint

#128

Post by itsTyrion »

theanonymousone wrote: post #9

May I ask if anyone has successfully used 1B and 3B models in production and if yes, in what use cases? I seem to be failing even in seemingly simpler tasks such as word translation or zero-shot classification. For example, they seem to not care about instructions to only write a response and no explanation, thus making it impossible to use them in a pipeline :/

I've tried using 3B outside of production. Asked it to be the character needed, like 30 words and use German. Instructions were consistently ignored, sometimes sentences devolved into Gibberish or English was mixed in halfway through. Don't even want to know how lobotomized 1B is.
Post reply on HN