Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

arxiv.org

Post reply on HN
Print view View original

21–30 of 61 posts

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#21

Post by _ache_ »

woadwarrior01 wrote: post #7
loufe wrote: post #2

It would be nice for the inference time to be paired with measure of output quality. I'm not well versed in how the architecture works, but I have a hard time believing a 90% reduction in peak memory footprint comes cost-free.

It's not cost-free. It comes at the cost of greatly increased latency. 29.9 seconds per token with Llama 3.1-70B. This is from Table 1 (pg 8) of the paper.

That is s/token and not token/s. The cost is high.

The actual goal of the article is to highlight that we can optimise the overall speed by decreasing link latency. Yeah link latency, because it's not 1 machine but several low devices that are used together to serve the 70B LLM.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#23

Post by _ache_ »

m3kw9 wrote: post #8
woadwarrior01 wrote: post #7

Earlier quoted context omitted.

It's not cost-free. It comes at the cost of greatly increased latency. 29.9 seconds per token with Llama 3.1-70B. This is from Table 1 (pg 8) of the paper.

Ah the disk swap method

It's not disk swap. It's multi-devices LLM.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#24

Post by vessenes »

This is not a memory reduction technique that's somehow magical. Well, it does manage memory with some clever scheduling. The core of this idea is that you can schedule out inference on edge nodes in a memory and bandwidth optimized way that's a bit different than just splitting layers.

They propose that right now computation and latency dominate the costs for multi-node inference, and pick a network topology (star) that is savvy to that.

That said, it's 26-29 seconds per token for llama2-70b with their 8 edge devices, each using 4 gigs of RAM. That's amazing that they can run it at all, but this isn't going to be viable at the edge with current hardware.

I think the paper makes the case that you could probably recruit say your 30 graphics workstations to do much faster inference without just nailing your LAN bandwidth, though.

Upshot - interesting paper -- smart ideas, large frontier models still need very exotic hardware and bandwidth interconnects - this may point a way forward to lowering the bandwidth interconnects part of the story.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#26

Post by sgc »

refulgentis wrote: post #10
freehorse wrote: post #5

Earlier quoted context omitted.

From what I get skimming through the article the main cost is speed of token generation (token latency). You can always run a large model by reading directly from the disk and not care much about ram; but it is very slow. They try to improve that aspect doing some optimisations, but it is still definitely slower than using ram or vram.

Table 3 directly refutes this* and claims 0 tradeoffs.** Below that, they indicate that a key part of the implementation is loading weights from disk before they're needed using a separate thread.*** * maybe I'm missing something though, someone please triple check :) ** ttft (time to first token) and s/token (seconds per token) are both lower than any alternative in all cases. *** "daemon thread asynchronously prelo…

I want to add that their chart shows s/token per device (edit: as per the heading on table 1 - it could also be confusing grammar), so it sounds like you are getting 4x the listed s/t on their 4 laptop cluster . Their laptops are not even hardwired - they are connecting over wifi.

This comes at a very interesting time for me. I have an ancient dual xeon workstation with 64gb memory that I was researching how to convert to run an llm. I can just run that with 4 instances on the same machine and see how it goes, without purchasing a better GPU, to start. It sounds like this will allow you to run very large models with minimal quants, on craigslist quality devices.

If it does what they say it does (and it seems to do), it will be an absolute game changer for most users.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#27

Post by o11c »

dvh wrote: post #11

So when will I be able to "sudo apt-get install llm" ?

Realistically, you probably want to wait until Vulkan support trickles out. That way, you aren't at the whim of the various evil hardware drivers (everybody's suck), and the AI can give you a disappointingly confused answer much faster than running the LLM on a CPU can.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#29

Post by tgtweak »

vessenes wrote: post #24

This is not a memory reduction technique that's somehow magical. Well, it does manage memory with some clever scheduling. The core of this idea is that you can schedule out inference on edge nodes in a memory and bandwidth optimized way that's a bit different than just splitting layers. They propose that right now computation and latency dominate the costs for multi-node inference, and pick a network topology (star)…

I think the main advantage here is you COULD run it, even it it takes a while. That is a step up from current model limitations which require ram or vram to hold the model.

I think this lays some groundwork for running a 400B model on a 3090/4090 or even smaller GPU. If you can get a huge model like that running on a single gpu even if the mean time per token is in the seconds, that's acceptable for many use cases.

If this same technique can be used to extend context windows in addition to token autocomplete, that would be great in it's own regard.

Hopefully work like this continues as throwing a ton of vram at a model should be regarded as a performance optimization not necessarily a requirement.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#30

Post by tgtweak »

zackangelo wrote: post #4
loufe wrote: post #2

It would be nice for the inference time to be paired with measure of output quality. I'm not well versed in how the architecture works, but I have a hard time believing a 90% reduction in peak memory footprint comes cost-free.

I've only read the abstract but they don't mention quantizing the weights or otherwise trying to shrink the model in any way. They're claiming to be able to efficiently run larger models without loading the entire thing into GPU memory. If they're using the same weights, the same architecture and just using tensor parallel operations to perform the forward pass that would imply no loss in quality. I'm sure there are…

I read it like this too - no drop in weights or model quality just optimizing the lower boundaries of performance when you are splitting from vram to ram to disk (or network).
Post reply on HN