Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

arxiv.org

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#31

Post by alchemist1e9 »

vessenes wrote: post #24

This is not a memory reduction technique that's somehow magical. Well, it does manage memory with some clever scheduling. The core of this idea is that you can schedule out inference on edge nodes in a memory and bandwidth optimized way that's a bit different than just splitting layers. They propose that right now computation and latency dominate the costs for multi-node inference, and pick a network topology (star)…

> I think the paper makes the case that you could probably recruit say your 30 graphics workstations to do much faster inference without just nailing your LAN bandwidth, though.

Could be a big deal if it allows cluster of smaller GPUs to compete with a single large VRAM GPU.

Unfortunately I’m a few months of date - which is an eternity in LLM inference techniques - so I’m not sure what current state of distributed inference looks like.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#32

Post by tgtweak »

Zetaphor wrote: post #25

Is this different from (or related to) the work being done by the exo project? https://github.com/exo-explore/exo

Exo is for partitioning over network across devices (implementing some bandwidth-reducing partitions) but still requires a minimum ram/vram requirement to load a model. This could, in theory, be combined to allow larger models to run on exo clusters with less gpu/ram than is required by the underlying model (at the cost of some performance no doubt, but still).

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#33

Post by kridsdale3 »

_ache_ wrote: post #23
m3kw9 wrote: post #8

Earlier quoted context omitted.

Ah the disk swap method

It's not disk swap. It's multi-devices LLM.

That looked like an analogy. Back in the days of a mechanical arm moving magnetic fields around in our PCs, you could have the illusion of infinite RAM as long as you're ok with microsecond operations now taking two million times longer. This is akin.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#34

Post by wpietri »

_ache_ wrote: post #23
m3kw9 wrote: post #8

Earlier quoted context omitted.

Ah the disk swap method

It's not disk swap. It's multi-devices LLM.

I think the point is that it has the same sort of latency tradeoff that disk swap did: it's awful, but sometimes better than nothing.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#35

Post by teraflop »

woadwarrior01 wrote: post #7
loufe wrote: post #2

It would be nice for the inference time to be paired with measure of output quality. I'm not well versed in how the architecture works, but I have a hard time believing a 90% reduction in peak memory footprint comes cost-free.

It's not cost-free. It comes at the cost of greatly increased latency. 29.9 seconds per token with Llama 3.1-70B. This is from Table 1 (pg 8) of the paper.

Am I just misunderstanding, or is the paper using "latency" when what they really mean is "throughput"?

In other words, if I want 100 tokens of output, do I have to wait 2990 seconds? If so, the terminology seems unnecessarily confusing.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#36

Post by michaelt »

tgtweak wrote: post #29
vessenes wrote: post #24

This is not a memory reduction technique that's somehow magical. Well, it does manage memory with some clever scheduling. The core of this idea is that you can schedule out inference on edge nodes in a memory and bandwidth optimized way that's a bit different than just splitting layers. They propose that right now computation and latency dominate the costs for multi-node inference, and pick a network topology (star)…

I think the main advantage here is you COULD run it, even it it takes a while. That is a step up from current model limitations which require ram or vram to hold the model. I think this lays some groundwork for running a 400B model on a 3090/4090 or even smaller GPU. If you can get a huge model like that running on a single gpu even if the mean time per token is in the seconds, that's acceptable for many use cases. I…

It's already technically possible to run huge models locally when you don't have the RAM/VRAM needed - llama.cpp can 'mmap' the model from disk.

Of course an nvidia 4090 has a memory bandwidth of a 1000 GB per second; a CPU like the i7-13700K has a memory bandwidth of 90 GB per second; and a high-end NVMe SSD might only have read bandwidth of 10 GB per second.

So in approximate terms, an LLM and quantisation level that can produce 10 tokens per second on a 4090 will produce 1 token per second from RAM and a token every 10 seconds from SSD.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#37

Post by ignoramous »

tgtweak wrote: post #29
vessenes wrote: post #24

This is not a memory reduction technique that's somehow magical. Well, it does manage memory with some clever scheduling. The core of this idea is that you can schedule out inference on edge nodes in a memory and bandwidth optimized way that's a bit different than just splitting layers. They propose that right now computation and latency dominate the costs for multi-node inference, and pick a network topology (star)…

I think the main advantage here is you COULD run it, even it it takes a while. That is a step up from current model limitations which require ram or vram to hold the model. I think this lays some groundwork for running a 400B model on a 3090/4090 or even smaller GPU. If you can get a huge model like that running on a single gpu even if the mean time per token is in the seconds, that's acceptable for many use cases. I…

> That is a step up from current model limitations which require ram or vram to hold the model.

Current? Apple recently published a neat paper on how they optimise for both inference (cpu/gpu) and memory use:

  Our method involves constructing an inference cost model that takes into account
  the characteristics of flash memory, guiding us to optimize in two critical areas:
  reducing the volume of data transferred from flash and reading data in larger, more contiguous
  chunks. Within this hardware-informed framework, we introduce two principal techniques.
  First, “windowing” strategically reduces data transfer by reusing previously activated neurons,
  and second, “row-column bundling”, tailored to the sequential data access strengths
  of flash memory, increases the size of data chunks read from flash memory. These methods
  collectively enable running models up to twice the size of the available DRAM, with
  up to 4x and 20x increase in inference speed compared to naive loading approaches in CPU
  and GPU, respectively.
https://news.ycombinator.com/item?id=38704982

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#38

Post by adam_arthur »

While I do think there's going to be a huge market for cloud-based LLM serving, the fact that consumer hardware can run close to SOTA models fairly easily (e.g. high-RAM MBP config), seems to me that the provider market won't be as big as investors are betting on.

Most of the rewards will be reaped by consumers rather than providers.

We're also in an age where the current levels of RAM in consumer devices were almost entirely optimized for prior to the existence of LLMs. I find it highly likely vendors will optimize for higher RAM capacity over other priorities in future hardware.

How long until a 256GB RAM laptop (shared with GPU) is reasonably cheap/available? I give it a few years at most.

It's possible that models grow orders of magnitude larger, but I find it more likely that the size of models will grow along the curve of cost of training decreasing/hardware cost improvements. There will be a sweet spot where it's economical to train larger models, and private companies won't push much beyond that.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#39

Post by k1musab1 »

vessenes wrote: post #24

This is not a memory reduction technique that's somehow magical. Well, it does manage memory with some clever scheduling. The core of this idea is that you can schedule out inference on edge nodes in a memory and bandwidth optimized way that's a bit different than just splitting layers. They propose that right now computation and latency dominate the costs for multi-node inference, and pick a network topology (star)…

Do you think this could allow distributed inference only, or opens the door for distributed training of the model? Democratization of the models is in part hampered by the total compute a single person or small group can make use of, but if a project like folding@home, but for training large models is possible, it could change the game somewhat.

Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]

#40

Post by lxgr »

adam_arthur wrote: post #38

While I do think there's going to be a huge market for cloud-based LLM serving, the fact that consumer hardware can run close to SOTA models fairly easily (e.g. high-RAM MBP config), seems to me that the provider market won't be as big as investors are betting on. Most of the rewards will be reaped by consumers rather than providers. We're also in an age where the current levels of RAM in consumer devices were almost…

Enterprises use LLMs too, and quite often there wouldn't be any client you could reasonably run the model on. (You wouldn't want to e.g. have an LLM summarize and categorize a user request on their device, since that would require you shipping your model and/or internal knowledge base to the client).
Post reply on HN