Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AirLLM 70B inference with single 4GB GPU

github.com

Post reply on HN
Print view View original

31–40 of 92 posts

Re: AirLLM 70B inference with single 4GB GPU

#31

Post by hiramwen »

titularcomment wrote: post #24
hiramwen wrote: post #8

Earlier quoted context omitted.

You don't really need a maintainer when codex or claude code can set it up for you; thats how I got Trellis2 working on windows and tiny VRAM despite Microsoft recommending you have 24GB VRAM and Linux. Models are pretty disposable now.

This is complete dependency on LLM tools all the way from development to usage, and is risky as well as prone to failure

Local inference without access to internet and tools is pretty safe, worst you get is a slop output, best case it solves your prompt.

Re: AirLLM 70B inference with single 4GB GPU

#32

Post by reactordev »

bensyverson wrote: post #28
pizza234 wrote: post #18

Earlier quoted context omitted.

> but are they functionally useful or helpful to anyone currently? Yes and no, depend on your expectations. Some/many like to run local LLMs just for the sake of it, so anything will do. MoE are useful on PC systems, at the condition of having high enough memory bandwidth (and large amounts of RAM) - that is, Threadripper/Pro. The advantage of MoE is that only a subset of the model's experts is used for each token, s…

Are people getting decent tokens/second throughput? Some of these demos crawl at 1 tok/s or worse, which limits their utility.

yes, I average 80-120 tok/s on my RTX 3080 with gemma 4 and faster with Qwen 3.5. The main use-case here is just code-monkey agents. I'm not looking for architectural guidance, but an agent to take a spec and complete it.

Re: AirLLM 70B inference with single 4GB GPU

#33

Post by Foobar8568 »

akie wrote: post #19
roger_ wrote: post #6

Seeing a lot of these “run 1TB models with 1GB RAM” projects recently. Most seem vibe coded and probably won’t be maintained. Hoping a winner emerges with some real momentum behind it.

No no, don't just say "vibe coded", say "Fable and $500 of credits"

Opus 4.6 was already enough to tackle these projects vibe coding.

Re: AirLLM 70B inference with single 4GB GPU

#34

Post by bensyverson »

reactordev wrote: post #32
bensyverson wrote: post #28

Earlier quoted context omitted.

Are people getting decent tokens/second throughput? Some of these demos crawl at 1 tok/s or worse, which limits their utility.

yes, I average 80-120 tok/s on my RTX 3080 with gemma 4 and faster with Qwen 3.5. The main use-case here is just code-monkey agents. I'm not looking for architectural guidance, but an agent to take a spec and complete it.

And is this using conventional model loading (all in VRAM), or are you streaming it in some way?

Re: AirLLM 70B inference with single 4GB GPU

#36

Post by gwerbin »

Foobar8568 wrote: post #33
akie wrote: post #19

Earlier quoted context omitted.

No no, don't just say "vibe coded", say "Fable and $500 of credits"

Opus 4.6 was already enough to tackle these projects vibe coding.

Incidentally I think I was more productive with Opus 4.6 than with any subsequent Anthropic model.

Re: AirLLM 70B inference with single 4GB GPU

#38

Post by jedbrooke »

cpfohl wrote: post #13
dofm wrote: post #5

Earlier quoted context omitted.

If you point it at a huggingface model identifier, it will download it, I assume. No way around that. It reads like it is keeping only the core and the active layer loaded at any one point, and streams layers from disk; there are several other solutions like this and if my understanding is right, this is probably better than an mmap implementation or just streaming experts in.

Obviously needs downloading eventually :). It seems like this tool saves on both disk space and RAM, then. Classic trade off: speed vs space.

I mean, if you had fast enough internet connection, you could just stream it over https

Re: AirLLM 70B inference with single 4GB GPU

#39

Post by tecleandor »

hiramwen wrote: post #8
roger_ wrote: post #6

Seeing a lot of these “run 1TB models with 1GB RAM” projects recently. Most seem vibe coded and probably won’t be maintained. Hoping a winner emerges with some real momentum behind it.

You don't really need a maintainer when codex or claude code can set it up for you; thats how I got Trellis2 working on windows and tiny VRAM despite Microsoft recommending you have 24GB VRAM and Linux. Models are pretty disposable now.

So, instead of having one or two maintained projects that work super well, now we have hundreds or thousands of people half-assing it locally each time they need it? Doesn't sound efficient.
Post reply on HN