Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

arxiv.org

Post reply on HN
Print view View original

41–50 of 61 posts

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#41

Post by adam_arthur »

lxgr wrote: post #40
adam_arthur wrote: post #38

While I do think there's going to be a huge market for cloud-based LLM serving, the fact that consumer hardware can run close to SOTA models fairly easily (e.g. high-RAM MBP config), seems to me that the provider market won't be as big as investors are betting on. Most of the rewards will be reaped by consumers rather than providers. We're also in an age where the current levels of RAM in consumer devices were almost…

Enterprises use LLMs too, and quite often there wouldn't be any client you could reasonably run the model on. (You wouldn't want to e.g. have an LLM summarize and categorize a user request on their device, since that would require you shipping your model and/or internal knowledge base to the client).

Yes, but if you can run a sufficient LLM on a $2,000 laptop, then the cost to serve it from the cloud will be similarly cheap. (e.g. reserve an appropriately sized EC2 instance for pennies on the dollar)

It's a highly competitive market. Companies aren't going to pay 100k/year to run a model on something that can run on a 2k consumer grade device.

128GB of gpu accessible/fast RAM can be had for $5000 on a macbook pro today. What will it be 3-4 years from now on linux/windows machines?

And we still haven't seen any SoC providers try to optimize for RAM capacity over compute yet.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#42

Post by Palomides »

alchemist1e9 wrote: post #31
vessenes wrote: post #24

This is not a memory reduction technique that's somehow magical. Well, it does manage memory with some clever scheduling. The core of this idea is that you can schedule out inference on edge nodes in a memory and bandwidth optimized way that's a bit different than just splitting layers. They propose that right now computation and latency dominate the costs for multi-node inference, and pick a network topology (star)…

> I think the paper makes the case that you could probably recruit say your 30 graphics workstations to do much faster inference without just nailing your LAN bandwidth, though. Could be a big deal if it allows cluster of smaller GPUs to compete with a single large VRAM GPU. Unfortunately I’m a few months of date - which is an eternity in LLM inference techniques - so I’m not sure what current state of distributed in…

llama.cpp supports splitting work across multiple nodes on a network already

it essentially just copies a chunk of the model to each one, works well for situations where each machine has limited vram

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#43

Post by simsla »

lxgr wrote: post #40
adam_arthur wrote: post #38

While I do think there's going to be a huge market for cloud-based LLM serving, the fact that consumer hardware can run close to SOTA models fairly easily (e.g. high-RAM MBP config), seems to me that the provider market won't be as big as investors are betting on. Most of the rewards will be reaped by consumers rather than providers. We're also in an age where the current levels of RAM in consumer devices were almost…

Enterprises use LLMs too, and quite often there wouldn't be any client you could reasonably run the model on. (You wouldn't want to e.g. have an LLM summarize and categorize a user request on their device, since that would require you shipping your model and/or internal knowledge base to the client).

Depends, shipping part of it (just an encoder or decoder) could still work.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#44

Post by alexandercheema »

tgtweak wrote: post #32
Zetaphor wrote: post #25

Is this different from (or related to) the work being done by the exo project? https://github.com/exo-explore/exo

Exo is for partitioning over network across devices (implementing some bandwidth-reducing partitions) but still requires a minimum ram/vram requirement to load a model. This could, in theory, be combined to allow larger models to run on exo clusters with less gpu/ram than is required by the underlying model (at the cost of some performance no doubt, but still).

exo maintainer here. tgtweak is correct.

This looks like potentially some promising research that I'm looking into reproducing now. We want to lower the barrier to running large models as much as possible so if this works, it would be a potential addition to the exo offering.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#45

Post by lxgr »

adam_arthur wrote: post #41
lxgr wrote: post #40

Earlier quoted context omitted.

Enterprises use LLMs too, and quite often there wouldn't be any client you could reasonably run the model on. (You wouldn't want to e.g. have an LLM summarize and categorize a user request on their device, since that would require you shipping your model and/or internal knowledge base to the client).

Yes, but if you can run a sufficient LLM on a $2,000 laptop, then the cost to serve it from the cloud will be similarly cheap. (e.g. reserve an appropriately sized EC2 instance for pennies on the dollar) It's a highly competitive market. Companies aren't going to pay 100k/year to run a model on something that can run on a 2k consumer grade device. 128GB of gpu accessible/fast RAM can be had for $5000 on a macbook pro…

Oh yes, I could definitely see the privacy-preserving consumer use case creating sufficient demand for efficiency that also bleeds over into the enterprise market.

That's what's happened with power efficiency and ARM CPUs, after all!

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#46

Post by lxgr »

simsla wrote: post #43
lxgr wrote: post #40

Earlier quoted context omitted.

Enterprises use LLMs too, and quite often there wouldn't be any client you could reasonably run the model on. (You wouldn't want to e.g. have an LLM summarize and categorize a user request on their device, since that would require you shipping your model and/or internal knowledge base to the client).

Depends, shipping part of it (just an encoder or decoder) could still work.

Even if bandwidth weren't an issue and all users had compatible hardware: You'd still be offloading a (semi-)trusted computation to user hardware, which is usually completely untrusted.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#47

Post by diggan »

tgtweak wrote: post #29
vessenes wrote: post #24

This is not a memory reduction technique that's somehow magical. Well, it does manage memory with some clever scheduling. The core of this idea is that you can schedule out inference on edge nodes in a memory and bandwidth optimized way that's a bit different than just splitting layers. They propose that right now computation and latency dominate the costs for multi-node inference, and pick a network topology (star)…

I think the main advantage here is you COULD run it, even it it takes a while. That is a step up from current model limitations which require ram or vram to hold the model. I think this lays some groundwork for running a 400B model on a 3090/4090 or even smaller GPU. If you can get a huge model like that running on a single gpu even if the mean time per token is in the seconds, that's acceptable for many use cases. I…

> I think the main advantage here is you COULD run it, even it it takes a while.

I mean, you COULD run it before as well, even if you don't have enough RAM or VRAM, by using something like `zram`. It'd probably be even slower (and border-line usable, depending on the use case), but it's not impossible to get things to run.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#48

Post by adunsulag »

lxgr wrote: post #45
adam_arthur wrote: post #41

Earlier quoted context omitted.

Yes, but if you can run a sufficient LLM on a $2,000 laptop, then the cost to serve it from the cloud will be similarly cheap. (e.g. reserve an appropriately sized EC2 instance for pennies on the dollar) It's a highly competitive market. Companies aren't going to pay 100k/year to run a model on something that can run on a 2k consumer grade device. 128GB of gpu accessible/fast RAM can be had for $5000 on a macbook pro…

Oh yes, I could definitely see the privacy-preserving consumer use case creating sufficient demand for efficiency that also bleeds over into the enterprise market. That's what's happened with power efficiency and ARM CPUs, after all!

This is where I want highly sensitive healthcare consumers of LLMs to be at. Note summation, suggested diagnosis (provider always in control), and other augmented abilities for the clinical staff without the risk of health care data sent outside the device, or the very local network.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#49

Post by adam_arthur »

lxgr wrote: post #45
adam_arthur wrote: post #41

Earlier quoted context omitted.

Yes, but if you can run a sufficient LLM on a $2,000 laptop, then the cost to serve it from the cloud will be similarly cheap. (e.g. reserve an appropriately sized EC2 instance for pennies on the dollar) It's a highly competitive market. Companies aren't going to pay 100k/year to run a model on something that can run on a 2k consumer grade device. 128GB of gpu accessible/fast RAM can be had for $5000 on a macbook pro…

Oh yes, I could definitely see the privacy-preserving consumer use case creating sufficient demand for efficiency that also bleeds over into the enterprise market. That's what's happened with power efficiency and ARM CPUs, after all!

Not sure what you mean: https://aws.amazon.com/ec2/graviton/

Not to speak of managed cloud services that run on ARM under-the-hood/behind the scenes.

Of course ARM isn't inherently cheaper, AMD+Intel could cut prices/margins big and probably be competitive on $/perf

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#50

Post by [deleted] »

vessenes wrote: post #24

This is not a memory reduction technique that's somehow magical. Well, it does manage memory with some clever scheduling. The core of this idea is that you can schedule out inference on edge nodes in a memory and bandwidth optimized way that's a bit different than just splitting layers. They propose that right now computation and latency dominate the costs for multi-node inference, and pick a network topology (star)…

[deleted]
Post reply on HN