Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AirLLM 70B inference with single 4GB GPU

github.com

Post reply on HN
Print view View original

71–80 of 92 posts

Re: AirLLM 70B inference with single 4GB GPU

#71

Post by sixothree »

xg15 wrote: post #51
imenani wrote: post #4

For anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token https://github.com/lyogavin/airllm/releases/tag/v3.1.0

At that point, how does this compare with simply running the model on the CPU?

Not an answer to your question, but maybe this has more info? I think with these optimizations and quants, it compares negatively. But can these optimizations be applied to models you want? Another question. https://news.ycombinator.com/item?id=48353348

Re: AirLLM 70B inference with single 4GB GPU

#72

Post by tecleandor »

serf wrote: post #66
tecleandor wrote: post #39

Earlier quoted context omitted.

So, instead of having one or two maintained projects that work super well, now we have hundreds or thousands of people half-assing it locally each time they need it? Doesn't sound efficient.

you're right, but flipped on its' head it also means that a well-meaning FOSS project may not ship out a zero-day in the future that affects half the planet at once. now we'll just always be vulnerable from different angles in different and unique ways across the globe. better? well, no.. but it is different, and biology has offered tons of wisdom about why it's a good thing to achieve things different ways across th…

There has to be a middle ground between a monoculture and not sharing a single knowledge bit between every human. Because if everyone is making their half assed thing just for themselves using LLM without sharing anything, that also means FOSS will stop existing, and knowledge will only evolve inside the corporations that have money for creating their provate knowledge bases.

Re: AirLLM 70B inference with single 4GB GPU

#73

Post by hatthew »

I've been seeing a lot of projects that let one use large models on machines with small amounts of memory. They seem to all be doing significant quantization and/or expert streaming. What's the benefit of these projects over something like taking an unsloth quant and running llama.cpp with appropriate flags (-cmoe/-mmap) to manage VRAM vs RAM vs SSD?

Re: AirLLM 70B inference with single 4GB GPU

#74

Post by senectus1 »

malshe wrote: post #63
imenani wrote: post #4

For anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token https://github.com/lyogavin/airllm/releases/tag/v3.1.0

I hope I'm not the only one who misread it as 292 tokens/s and got excited momentarily

haha its something i look out for now. having seen a few recent stories like this.

Re: AirLLM 70B inference with single 4GB GPU

#75

Post by killingtime74 »

root-parent wrote: post #48
imenani wrote: post #4

For anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token https://github.com/lyogavin/airllm/releases/tag/v3.1.0

It matches my coding speed...its ok.

it takes you almost 5 mins to write at most 1 word?

Re: AirLLM 70B inference with single 4GB GPU

#76

Post by 361994752 »

malshe wrote: post #63
imenani wrote: post #4

For anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token https://github.com/lyogavin/airllm/releases/tag/v3.1.0

I hope I'm not the only one who misread it as 292 tokens/s and got excited momentarily

I thought I met some huge breakthrough on computer science until I saw your comment

Re: AirLLM 70B inference with single 4GB GPU

#77

Post by SwellJoe »

mrwaip wrote: post #44

What device do I need and how much will it cost to install one at home so that it works as quickly as the Claude Code answer (and it answers quite slowly)?

If you want it to respond in under a minute, you need more hardware than this application is intended for. This thing's response is measured in seconds per token, not tokens per second.

To get Claude Code responsiveness from even a pretty small (but still usable) model, you need, maybe two 32GB GPUs? I run Gemma 4 31B and Qwen 3.6 27B on my dual 32GB GPU setup (cheap old Radeon Pro V620 GPUs) at about 20 t/s, which is not fast enough for comfortable interactive agentic use. A couple of new GPUs, like Radeon AI Pro 9700 at $1400 each, probably gets you fast enough for comfortable interactive use with small models like those. Those small models are not competitive with Claude models, however (maybe they beat Haiku sometimes). They can write a little Python or make a web page, they can't architect a real application.

To run a near-frontier model like Kimi K3 or GLM 5.2 at comfortable speeds, you need serious hardware with 768GB VRAM, minimum. I think Asus is releasing something like that for about $150k soon. You can run DeepSeek V4 Flash at almost comfortable speeds and in a decently capable quantization on two DGX Sparks or Asus GX10s (about $10,000).

Or, you could use DeepSeek V4 Flash from DeepSeek.com, at blistering speeds and with huge contexts, for something like a decade or two for that same $10,000.

Re: AirLLM 70B inference with single 4GB GPU

#80

Post by xlayn »

imenani wrote: post #4

For anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token https://github.com/lyogavin/airllm/releases/tag/v3.1.0

I have see a lot of "run using cpu" or using 1gb of ram... and the short answer is sure, you can run it in 1mb of ram, or in a 286, it will just take a couple of years to produce the first token.

the canary in the coal that this has not change is nvidia share price.

Post reply on HN