Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

arxiv.org

Post reply on HN
Print view View original

51–60 of 61 posts

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#51

Post by vessenes »

alchemist1e9 wrote: post #31
vessenes wrote: post #24

This is not a memory reduction technique that's somehow magical. Well, it does manage memory with some clever scheduling. The core of this idea is that you can schedule out inference on edge nodes in a memory and bandwidth optimized way that's a bit different than just splitting layers. They propose that right now computation and latency dominate the costs for multi-node inference, and pick a network topology (star)…

> I think the paper makes the case that you could probably recruit say your 30 graphics workstations to do much faster inference without just nailing your LAN bandwidth, though. Could be a big deal if it allows cluster of smaller GPUs to compete with a single large VRAM GPU. Unfortunately I’m a few months of date - which is an eternity in LLM inference techniques - so I’m not sure what current state of distributed in…

Yeah, I think these methods could be baked into llama.cpp or some other higher up the toolchain python library or what have you. They shard out each layer (ish?) to the edges, and recombine that layer inference at the master node, while the outside edges load up their next bit if they need to; I would guess the devil is in the details for all the possible tensor types and architectures (for instance, how shall we implement skip layers?).

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#52

Post by ay »

Palomides wrote: post #42
alchemist1e9 wrote: post #31

Earlier quoted context omitted.

> I think the paper makes the case that you could probably recruit say your 30 graphics workstations to do much faster inference without just nailing your LAN bandwidth, though. Could be a big deal if it allows cluster of smaller GPUs to compete with a single large VRAM GPU. Unfortunately I’m a few months of date - which is an eternity in LLM inference techniques - so I’m not sure what current state of distributed in…

llama.cpp supports splitting work across multiple nodes on a network already it essentially just copies a chunk of the model to each one, works well for situations where each machine has limited vram

Any pointers to RTFM / llama repo for that ? I could not find anything on a cursory look. Thanks in advance !

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#53

Post by Palomides »

ay wrote: post #52
Palomides wrote: post #42

Earlier quoted context omitted.

llama.cpp supports splitting work across multiple nodes on a network already it essentially just copies a chunk of the model to each one, works well for situations where each machine has limited vram

Any pointers to RTFM / llama repo for that ? I could not find anything on a cursory look. Thanks in advance !

https://github.com/ggerganov/llama.cpp/tree/master/examples/...

you run that on the remote nodes

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#54

Post by slimsag »

michaelt wrote: post #36
tgtweak wrote: post #29

Earlier quoted context omitted.

I think the main advantage here is you COULD run it, even it it takes a while. That is a step up from current model limitations which require ram or vram to hold the model. I think this lays some groundwork for running a 400B model on a 3090/4090 or even smaller GPU. If you can get a huge model like that running on a single gpu even if the mean time per token is in the seconds, that's acceptable for many use cases. I…

It's already technically possible to run huge models locally when you don't have the RAM/VRAM needed - llama.cpp can 'mmap' the model from disk. Of course an nvidia 4090 has a memory bandwidth of a 1000 GB per second; a CPU like the i7-13700K has a memory bandwidth of 90 GB per second; and a high-end NVMe SSD might only have read bandwidth of 10 GB per second. So in approximate terms, an LLM and quantisation level th…

I haven't been able to get llama.cpp's mmap logic to work on macOS

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#55

Post by tgtweak »

alexandercheema wrote: post #44
tgtweak wrote: post #32

Earlier quoted context omitted.

Exo is for partitioning over network across devices (implementing some bandwidth-reducing partitions) but still requires a minimum ram/vram requirement to load a model. This could, in theory, be combined to allow larger models to run on exo clusters with less gpu/ram than is required by the underlying model (at the cost of some performance no doubt, but still).

exo maintainer here. tgtweak is correct. This looks like potentially some promising research that I'm looking into reproducing now. We want to lower the barrier to running large models as much as possible so if this works, it would be a potential addition to the exo offering.

Yeah combining these two would make a lot of sense, there is a big appetite to run larger models - even slower - on clustered hardware. This way you can add compute to speed up the token pace vs adding it just to run the model at all.

It is also possible some of these optimizations could help optimize distribution based on latency and bandwidth between nodes.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#56

Post by tgtweak »

ignoramous wrote: post #37
tgtweak wrote: post #29

Earlier quoted context omitted.

I think the main advantage here is you COULD run it, even it it takes a while. That is a step up from current model limitations which require ram or vram to hold the model. I think this lays some groundwork for running a 400B model on a 3090/4090 or even smaller GPU. If you can get a huge model like that running on a single gpu even if the mean time per token is in the seconds, that's acceptable for many use cases. I…

> That is a step up from current model limitations which require ram or vram to hold the model. Current? Apple recently published a neat paper on how they optimise for both inference (cpu/gpu) and memory use: Our method involves constructing an inference cost model that takes into account the characteristics of flash memory, guiding us to optimize in two critical areas: reducing the volume of data transferred from fl…

Similar but I think the apple approach here requires model modification whereas afaik OPs solution works with the model verbatim. I could be wrong as I haven't looked into the code but given the specificity of the first article regarding hardware and model, I would assume that.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#57

Post by tgtweak »

michaelt wrote: post #36
tgtweak wrote: post #29

Earlier quoted context omitted.

I think the main advantage here is you COULD run it, even it it takes a while. That is a step up from current model limitations which require ram or vram to hold the model. I think this lays some groundwork for running a 400B model on a 3090/4090 or even smaller GPU. If you can get a huge model like that running on a single gpu even if the mean time per token is in the seconds, that's acceptable for many use cases. I…

It's already technically possible to run huge models locally when you don't have the RAM/VRAM needed - llama.cpp can 'mmap' the model from disk. Of course an nvidia 4090 has a memory bandwidth of a 1000 GB per second; a CPU like the i7-13700K has a memory bandwidth of 90 GB per second; and a high-end NVMe SSD might only have read bandwidth of 10 GB per second. So in approximate terms, an LLM and quantisation level th…

Yeah mmap and leaving the caching up to the OS or storage drivers instead of model-informed placement is not going to yield the same results. Observing data access patterns, pre-empting inference data requirements and understanding hardware latency when distributing the model can yield some pretty significant results, as that same approach does in other domains. This is adjacent to compiler optimization.

It would be very meta to use AI to observe these access patterns and distribute the model accordingly based on usage, so that it can optimize placement for your given context domain.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#58

Post by prometheon1 »

diggan wrote: post #47
tgtweak wrote: post #29

Earlier quoted context omitted.

I think the main advantage here is you COULD run it, even it it takes a while. That is a step up from current model limitations which require ram or vram to hold the model. I think this lays some groundwork for running a 400B model on a 3090/4090 or even smaller GPU. If you can get a huge model like that running on a single gpu even if the mean time per token is in the seconds, that's acceptable for many use cases. I…

> I think the main advantage here is you COULD run it, even it it takes a while. I mean, you COULD run it before as well, even if you don't have enough RAM or VRAM, by using something like `zram`. It'd probably be even slower (and border-line usable, depending on the use case), but it's not impossible to get things to run .

Zram compresses part of the data in ram, right? Can an LLM be compressed?

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#59

Post by xena »

prometheon1 wrote: post #58
diggan wrote: post #47

Earlier quoted context omitted.

> I think the main advantage here is you COULD run it, even it it takes a while. I mean, you COULD run it before as well, even if you don't have enough RAM or VRAM, by using something like `zram`. It'd probably be even slower (and border-line usable, depending on the use case), but it's not impossible to get things to run .

Zram compresses part of the data in ram, right? Can an LLM be compressed?

Not really. The weights are mostly random numbers.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#60

Post by tonetegeatinst »

While training seems to be out of reach for the average tech user unless they have a data center for a homelab or a very large income, SOTA models can be easily run on the edge devices either on a phone or a dedicated computer/server.

LocalLLAMA and the fact the open weights and open datasets have really helped show that these can be done if you have enough resources and motivation.

Post reply on HN