AirLLM 70B inference with single 4GB GPU
github.com
AirLLM 70B inference with single 4GB GPU
1–10 of 92 posts
Re: AirLLM 70B inference with single 4GB GPU
#2I’m still slightly confused on what this adds.
Let’s say I wanted to run a full size open weight model. I have a 128GB m3 max laptop.
Does this basically load layers in and out on demand? So I still have to download the full model to disk, but the RAM requirements go way down? The readme calls out that one still needs to connect HuggingFace, which leads me to believe that maybe you don’t even need to download the full model?
Re: AirLLM 70B inference with single 4GB GPU
#3[dead]
Re: AirLLM 70B inference with single 4GB GPU
#4For anyone wondering “how slow is this?”
IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token
Re: AirLLM 70B inference with single 4GB GPU
#5I’m still slightly confused on what this adds. Let’s say I wanted to run a full size open weight model. I have a 128GB m3 max laptop. Does this basically load layers in and out on demand? So I still have to download the full model to disk, but the RAM requirements go way down? The readme calls out that one still needs to connect HuggingFace, which leads me to believe that maybe you don’t even need to download the ful…
If you point it at a huggingface model identifier, it will download it, I assume. No way around that.
It reads like it is keeping only the core and the active layer loaded at any one point, and streams layers from disk; there are several other solutions like this and if my understanding is right, this is probably better than an mmap implementation or just streaming experts in.
Re: AirLLM 70B inference with single 4GB GPU
#6Seeing a lot of these “run 1TB models with 1GB RAM” projects recently. Most seem vibe coded and probably won’t be maintained.
Hoping a winner emerges with some real momentum behind it.
Re: AirLLM 70B inference with single 4GB GPU
#7For anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token https://github.com/lyogavin/airllm/releases/tag/v3.1.0
Ahaha thank you, I naively assumed the unlabeled graph in the readme was tps, not spt!
Re: AirLLM 70B inference with single 4GB GPU
#8Seeing a lot of these “run 1TB models with 1GB RAM” projects recently. Most seem vibe coded and probably won’t be maintained. Hoping a winner emerges with some real momentum behind it.
You don't really need a maintainer when codex or claude code can set it up for you; thats how I got Trellis2 working on windows and tiny VRAM despite Microsoft recommending you have 24GB VRAM and Linux. Models are pretty disposable now.
Re: AirLLM 70B inference with single 4GB GPU
#9I guess the use case is something like: you have a slightly obsolete Mac or PC or a whole bunch of them, and just need to compose one or more convincing spam emails, but it's fine if it takes a full week to do it?
Re: AirLLM 70B inference with single 4GB GPU
#10I guess the use case is something like: you have a slightly obsolete Mac or PC or a whole bunch of them, and just need to compose one or more convincing spam emails, but it's fine if it takes a full week to do it?
And you don’t pay for the electricity.