"The layer-by-layer streaming is clever.
Curious how the throughput compares to running a quantized model on the same
GPU — seems like quantization might still win on speed?"
AirLLM 70B inference with single 4GB GPU
81–90 of 92 posts
Re: AirLLM 70B inference with single 4GB GPU
#82Earlier quoted context omitted.
Does MoE help with multimodality? Can it in general enable reasoning in imagery (technical drawings, diagrams, schematics) rather than text-based?
MoE has nothing to do with multimodality. MoE is a concept proposed in 1991, before the deep learning era (which is before what I call the transformers era). You can think of it like sharing. Contrary to popular belief; 'experts' in MoE LLMs do not specialize. There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization. As for reasoning in non-text modalities, you…
> There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization.
Huh, always thought one would sort of require the other.
For a good MoE model, wouldn't I want to minimize the "churn" between experts, i.e. the amount of time one expert model has to be swapped in for another expert model? That would be naturally the way if experts correspond to semantic categories.
E.g. suppose I have a model that can answer questions in 100s of languages. Then while any of those languages might be requested by some caller, it's highly unlikely a caller will request all languages in the same session - realistically, there might be one or two languages in a session and those will then span the entire session.
There will also be languages that are requested very often and others that are extremely rare.
(Let's say my model also supports Klingon and Sindarin. Those are important for marketing reasons and because I genuinely like to make the occasional nerd happy - but practically, I get maybe a handful of requests for those every few months. So it would make sense to centralize the knowledge for those languages in some specific part of the model, so I can keep that part out of VRAM - and probably RAM as well - during the 99% of time where it's not needed)
So wouldn't it make sense to make the expert models language specific here? Then you could take advantage of the fact that a language rarely changes inside a session and keep that expert in VRAM for the entire session. You could also avoid dragging parameters along with you for languages that are practically never used.
Re: AirLLM 70B inference with single 4GB GPU
#83What device do I need and how much will it cost to install one at home so that it works as quickly as the Claude Code answer (and it answers quite slowly)?
If you want it to respond in under a minute, you need more hardware than this application is intended for. This thing's response is measured in seconds per token, not tokens per second. To get Claude Code responsiveness from even a pretty small (but still usable) model, you need, maybe two 32GB GPUs? I run Gemma 4 31B and Qwen 3.6 27B on my dual 32GB GPU setup (cheap old Radeon Pro V620 GPUs) at about 20 t/s, which i…
> two DGX Sparks or Asus GX10s (about $10,000)
> for something like a decade or two for that same $10,000
I've been using DS V4 Flash through OpenCode and it's mind-blowing how I get near-SOTA AI model for the price of two coffees.
But let's put the broken economy of AI APIs aside for a second. Running such a model with under 6 figures of fixed costs is equally mind-blowing. And we know that hardware costs are real, Nvidia is selling those at a profit. I'm not talking about consumers, but businesses. I remember paying much more for something more trivial, like Datadog.
Re: AirLLM 70B inference with single 4GB GPU
#84“Can run” and “is useful interactively” are different benchmarks. At this latency, I can still imagine batch or overnight jobs being interesting; for chat, time to first useful answer matters much more than whether the weights technically fit. A workload/latency/energy table would make projects like this easier to evaluate.
Re: AirLLM 70B inference with single 4GB GPU
#85Earlier quoted context omitted.
No no, don't just say "vibe coded", say "Fable and $500 of credits"
Opus 4.6 was already enough to tackle these projects vibe coding.
Yes, but with Fable they might actually work in the end. Opus is such trash, malicious compliance and overconfidence personified.
Re: AirLLM 70B inference with single 4GB GPU
#86For anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token https://github.com/lyogavin/airllm/releases/tag/v3.1.0
Are there benchmarks for smaller models?
Re: AirLLM 70B inference with single 4GB GPU
#87Earlier quoted context omitted.
If you want it to respond in under a minute, you need more hardware than this application is intended for. This thing's response is measured in seconds per token, not tokens per second. To get Claude Code responsiveness from even a pretty small (but still usable) model, you need, maybe two 32GB GPUs? I run Gemma 4 31B and Qwen 3.6 27B on my dual 32GB GPU setup (cheap old Radeon Pro V620 GPUs) at about 20 t/s, which i…
> two DGX Sparks or Asus GX10s (about $10,000) > for something like a decade or two for that same $10,000 I've been using DS V4 Flash through OpenCode and it's mind-blowing how I get near-SOTA AI model for the price of two coffees. But let's put the broken economy of AI APIs aside for a second. Running such a model with under 6 figures of fixed costs is equally mind-blowing. And we know that hardware costs are real,…
DeepSeek feels like the most...trying to think of the word, honest isn't quite right but close, of the AI companies. They make very good models that are phenomenally efficient and they sell them at what seems to be a modest margin. They're just running a business. Doesn't feel like a Ponzi scheme. It doesn't feel like they're trying to take over the world and be the only AI company through regulatory capture or cornering the market on RAM and compute, they're not adding "memory" and personality tuned to maximize psychosis or addiction. They're not benchmaxxing the way Kimi and Qwen are benchmaxxed; you can find the holes and weirdness in those models capabilities pretty quickly. DeepSeek models feel like good all-rounders. Not the best at anything, but you won't be shocked by it doing stupid shit, either. Qwen keeps stunning me with really dumb bugs it creates even while being able to work on really big really hard problems, it's surprisingly sloppy.
I dunno. I don't have good feelings about most of the AI companies; I feel like they're trying to be predatory and monopolistic. I don't get that feeling from DeepSeek, they seem like a decent company making a good product at a fair price. So, they're consistently my choice for API usage, even if I still use the best American models for agentic use, via a subscription.
Re: AirLLM 70B inference with single 4GB GPU
#88Earlier quoted context omitted.
It matches my coding speed...its ok.
it takes you almost 5 mins to write at most 1 word?
If you look at examples of complex code and based on a public NASA dataset:
https://raw.githubusercontent.com/timm/ourmine/refs/heads/ma...
that has 93 NASA software projects, you are looking at less than 9 LOC per engineer/day
9 LOC per engineer/day...yes its about 5 to 6 min per token. Do you do better?
Re: AirLLM 70B inference with single 4GB GPU
#89Re: AirLLM 70B inference with single 4GB GPU
#90For anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token https://github.com/lyogavin/airllm/releases/tag/v3.1.0
At that point, how does this compare with simply running the model on the CPU?
if you take a model that requires 200GB of VRAM and you run it on the CPU, it requires 200GB of RAM instead. Still unfeasible on consumer hardware. With this approach you can easily do it on 12GB or less of either RAM of VRAM, at several seconds per token instead of tokens per seconds. Very unusable, but certainly interesting!