Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Quantized Llama models with increased speed and a reduced memory footprint

ai.meta.com

Post reply on HN
Print view View original

111–120 of 128 posts

Re: Quantized Llama models with increased speed and a reduced memory footprint

#111

Post by tugdual »

yuvalr1 wrote: post #109

Looking at how to deploy 1B and 3B Llama models on Android for inference. Some posts online recommend using Termux (an amazing app) to have an emulated shell and then install as if it's Linux, using ollama for example. However, this forces you into a manual installation process, and also most of the people don't know what Termux is, and would be afraid to install it from F-Droid. Maybe someone can recommend a way to…

I actually did something similar using llama.cpp a while back, would be curious to see the speedup with this model.

https://github.com/TugdualKerjan/bunny/tree/main

Re: Quantized Llama models with increased speed and a reduced memory footprint

#112

Post by antonvs »

yuvalr1 wrote: post #109

Looking at how to deploy 1B and 3B Llama models on Android for inference. Some posts online recommend using Termux (an amazing app) to have an emulated shell and then install as if it's Linux, using ollama for example. However, this forces you into a manual installation process, and also most of the people don't know what Termux is, and would be afraid to install it from F-Droid. Maybe someone can recommend a way to…

This might be of use:

https://github.com/a-ghorbani/pocketpal-ai

Re: Quantized Llama models with increased speed and a reduced memory footprint

#113

Post by formalsystem »

qeternity wrote: post #96
cmsj wrote: post #94

It really bugs me that every time I see posts about new models, there is never any indication of how much VRAM one needs to actually run them.

That's because it's easily calculable and also somewhat impossible to say in any meaningful sense. Most weights are released as fp16/bf16 so 2 bytes per weight. So just double the number of parameters = the number of gigabytes of VRAM. Llama 3.1 8B ~= 16GB weights in fp16. At 4bit quantization, it would be half the number of parameters so Llama 3.1 8B ~= 4GB weights. But this is just weights. The real issue is contex…

You can estimate context length impact by doing back of the envelope calculations on KV cache size: 2 * layers * attention heads * head_dim * byte_per_element * batch_size * sequence_length

Some pretty charts here https://github.com/pytorch/ao/issues/539

Re: Quantized Llama models with increased speed and a reduced memory footprint

#114

Post by Evidlo »

formalsystem wrote: post #83
Evidlo wrote: post #75

Earlier quoted context omitted.

I have a non-ML question. In vanilla Pytorch I have the following expression: t.sum(values[inds] * weights) If 'inds' is int8, I get "IndexError: tensors used as indices must be long, int, byte or bool tensors". Is this still true if I use torchao?

The issue here is memory in PyTorch is byte addressable and that's a limitation we can't solve without making a lot more changes to PyTorch. But in your specific case, if you'd like to pack more data into `values` you can use a combination of clever bit shifting, torch.cat and other bit twiddling pytorch like ops to pack more data. It's a trick we use quite heavily in torchao

Arent int8s byte-aligned though? I thought this restriction was originally motivated by maintenance overhead of having to support more dtypes.

Re: Quantized Llama models with increased speed and a reduced memory footprint

#115

Post by [deleted] »

yuvalr1 wrote: post #109

Looking at how to deploy 1B and 3B Llama models on Android for inference. Some posts online recommend using Termux (an amazing app) to have an emulated shell and then install as if it's Linux, using ollama for example. However, this forces you into a manual installation process, and also most of the people don't know what Termux is, and would be afraid to install it from F-Droid. Maybe someone can recommend a way to…

[deleted]

Re: Quantized Llama models with increased speed and a reduced memory footprint

#116

Post by SoLoMo123 »

nikolayasdf123 wrote: post #61

what's your opinion on LlamaStack? for me it is nothing short of bad experience. it is way over-engineered with poor quality and just plain does not work, and maintainers are questionable. I would rather call HuggingFace python code for inference or anything else. is ExecuTorch any better?

Hi, I'm Mergen and I work on ExecuTorch.

ExecuTorch is a runtime for mobile and embedded devices to run PyTorch models directly. Currently it runs pretty fast on CPU, but expanding our use-case for mobile accelerators and GPUs.

We're still in our early stages (just turned beta status). But try it out and let us know.

Regarding Llama Stack, it is built by my colleagues. What were some concrete issues have you experienced? If you have error/bug reports, I'll happy to pass along.

Re: Quantized Llama models with increased speed and a reduced memory footprint

#117

Post by com2kid »

zackangelo wrote: post #108
com2kid wrote: post #40

Earlier quoted context omitted.

On my LinkedIn post about this topic someone actually replied with a superior method of steering LLM output compared to anything else I've ever heard of, so I've decided that until I find time to implement their method, I'm not going to worry about things. tl;dr you put into the prompt all the JSON up until what you want the LLM to say, and you set the stop token to the end token of the current JSON item (so ',' or '…

With mixlayer, because the round trip time to the model is so short, you can alternate between appending known tokens of the JSON output and values you want the model to generate. I think this works better than constraining the sampling in a lot of cases. We haven’t built a state machine over JSON schema that uses this approach yet but it’s on the way.

> With mixlayer, because the round trip time to the model is so short, you can alternate between appending known tokens of the JSON output and values you want the model to generate. I think this works better than constraining the sampling in a lot of cases.

Wow, that is a much more succinct way of describing it!

> We haven’t built a state machine over JSON schema that uses this approach yet but it’s on the way.

Really this should just be a simple library in JS and Python. Schema goes in, state machine pops out.

Complications will be around optional fields, I'm not sure offhand how to solve that!

Re: Quantized Llama models with increased speed and a reduced memory footprint

#118

Post by BoorishBears »

jkukul wrote: post #101
BoorishBears wrote: post #93

Earlier quoted context omitted.

Why not preprompt with ```json {

Yes, you can pre-fill the assistant's response with "```json {" or even "{" and that should increase the likelihood of getting a proper JSON in the response, but it's still not guaranteed. It's not nearly reliable enough for a production use case, even on a bigger (8B) model. I could recommend using ollama or VLLm inference servers. They support a `response_format="json"` parameter (by implementing grammars on top of…

Grammars are best but if you read their comment they're apparently using ollama in a situation that doesn't support them.

Re: Quantized Llama models with increased speed and a reduced memory footprint

#119

Post by zackangelo »

com2kid wrote: post #117
zackangelo wrote: post #108

Earlier quoted context omitted.

With mixlayer, because the round trip time to the model is so short, you can alternate between appending known tokens of the JSON output and values you want the model to generate. I think this works better than constraining the sampling in a lot of cases. We haven’t built a state machine over JSON schema that uses this approach yet but it’s on the way.

> With mixlayer, because the round trip time to the model is so short, you can alternate between appending known tokens of the JSON output and values you want the model to generate. I think this works better than constraining the sampling in a lot of cases. Wow, that is a much more succinct way of describing it! > We haven’t built a state machine over JSON schema that uses this approach yet but it’s on the way. Reall…

I'd love it if you checked out what we've been working on.

It's still in early stages, but might be usable for something you're trying to build. Here's an example (this buffers the entire JSON object, but you can also gen as you go): https://docs.mixlayer.com/examples/json-output

Re: Quantized Llama models with increased speed and a reduced memory footprint

#120

Post by kridsdale3 »

grahamj wrote: post #106
derefr wrote: post #58

Earlier quoted context omitted.

> But it's a pretty rare day at work that "apply a random rotation matrix to a 128-dimensional vector" is the solution to my problem. Funny enough, if you visualize a vector-embedding's latent-space features using that "points on the surface of a hypersphere" analogy that ML programmers like to use — and you assume a really low quantization, say, 1-bit — then you can almost picture the hypersphere surface as a black-…

I'm just on the edge of understanding this but if I'm visualizing this right you're talking about a point source at the center of a sphere and a bitmap indicating where all the vectors intersect the surface. But that would mean the lengths would all be the same. Isn't it the lengths/distances to neighbors that is the main information being stored in a vector db? Or is it just that what you're talking about only conce…

My understanding is that yes, it actually is normalized to have the lengths all be the same, and thus the angle from (hyperdimensional) 0,0,0,(...n) is all that matters. The "distance between two embeddings" is able to simply to cosign of the two angles.
Post reply on HN