For anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token https://github.com/lyogavin/airllm/releases/tag/v3.1.0
At that point, how does this compare with simply running the model on the CPU?
AirLLM 70B inference with single 4GB GPU
51–60 of 92 posts
Re: AirLLM 70B inference with single 4GB GPU
#52You can run any frontier model on your PC if you just wait long enough...
Re: AirLLM 70B inference with single 4GB GPU
#53Earlier quoted context omitted.
> but are they functionally useful or helpful to anyone currently? Yes and no, depend on your expectations. Some/many like to run local LLMs just for the sake of it, so anything will do. MoE are useful on PC systems, at the condition of having high enough memory bandwidth (and large amounts of RAM) - that is, Threadripper/Pro. The advantage of MoE is that only a subset of the model's experts is used for each token, s…
Are people getting decent tokens/second throughput? Some of these demos crawl at 1 tok/s or worse, which limits their utility.
in theory, QWEN3.6-35B-A3B should run as fast as a 3B model; and in practice, it can be just as dumb.
Re: AirLLM 70B inference with single 4GB GPU
#54Seeing a lot of these “run 1TB models with 1GB RAM” projects recently. Most seem vibe coded and probably won’t be maintained. Hoping a winner emerges with some real momentum behind it.
No no, don't just say "vibe coded", say "Fable and $500 of credits"
these things can be done with local models; so don't say 500$ credits, say "accelerated climate change disasters by $500/year"
Re: AirLLM 70B inference with single 4GB GPU
#55Re: AirLLM 70B inference with single 4GB GPU
#56Re: AirLLM 70B inference with single 4GB GPU
#57Earlier quoted context omitted.
Opus 4.6 was already enough to tackle these projects vibe coding.
Incidentally I think I was more productive with Opus 4.6 than with any subsequent Anthropic model.
Yea, opus 4.6 is much better than opus 5 because it is willing to do task. Opus 5 just gaslights and refuses to solve many complex problem.
Re: AirLLM 70B inference with single 4GB GPU
#58Earlier quoted context omitted.
If you point it at a huggingface model identifier, it will download it, I assume. No way around that. It reads like it is keeping only the core and the active layer loaded at any one point, and streams layers from disk; there are several other solutions like this and if my understanding is right, this is probably better than an mmap implementation or just streaming experts in.
Not an expert in this field, but the "expert" is consisting of multiple layers. To keep it small in terms of memory print, this project streams each layer (dividing even further). It also requires extra space because of decomposition of the layers. Normally the file format optimized for compute intense workloads. But here the bottleneck is the memory capacity. Also guessing that you need to be able to hold at least 3…
Thanks for the extra detail here.
Re: AirLLM 70B inference with single 4GB GPU
#59Earlier quoted context omitted.
Does MoE help with multimodality? Can it in general enable reasoning in imagery (technical drawings, diagrams, schematics) rather than text-based?
MoE has nothing to do with multimodality. MoE is a concept proposed in 1991, before the deep learning era (which is before what I call the transformers era). You can think of it like sharing. Contrary to popular belief; 'experts' in MoE LLMs do not specialize. There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization. As for reasoning in non-text modalities, you…
> You can think of it like sharing.
was this meant to read "sharding"?
Re: AirLLM 70B inference with single 4GB GPU
#60[flagged]