Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AirLLM 70B inference with single 4GB GPU

github.com

Post reply on HN
Print view View original

41–50 of 92 posts

Re: AirLLM 70B inference with single 4GB GPU

#41

Post by parineum »

nickpsecurity wrote: post #30
bensyverson wrote: post #14

Earlier quoted context omitted.

I do think these “run a bigger model than will fit in VRAM” projects are necessary steps, but are they functionally useful or helpful to anyone currently? For example, is anyone out there running a big Qwen for coding on a 16-32GB machine with these techniques?

If I could justify wear and tear and electricity, I was willing to do something like this for batch processing. The batches would be a bunch of prompts whose outputs I'd look at the next day. Maybe common operations, like QA or refactoring, on whatever software I wrote. If so, I could use a larger model than I have real-time hardware for. The largest, well-trained models can often get the output mostly right in one t…

at 5 minutes per token, you could look at the results next week

Re: AirLLM 70B inference with single 4GB GPU

#42

Post by [deleted] »

bensyverson wrote: post #14
imenani wrote: post #4

For anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token https://github.com/lyogavin/airllm/releases/tag/v3.1.0

I do think these “run a bigger model than will fit in VRAM” projects are necessary steps, but are they functionally useful or helpful to anyone currently? For example, is anyone out there running a big Qwen for coding on a 16-32GB machine with these techniques?

[deleted]

Re: AirLLM 70B inference with single 4GB GPU

#43

Post by dummydummy1234 »

dannyw wrote: post #29
ddevnyc wrote: post #27

Earlier quoted context omitted.

Does MoE help with multimodality? Can it in general enable reasoning in imagery (technical drawings, diagrams, schematics) rather than text-based?

MoE has nothing to do with multimodality. MoE is a concept proposed in 1991, before the deep learning era (which is before what I call the transformers era). You can think of it like sharing. Contrary to popular belief; 'experts' in MoE LLMs do not specialize. There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization. As for reasoning in non-text modalities, you…

Wait I thought the router ends up specializing the experts?

Like there is no explicit goal aside from each 'expert' getting roughly equal weight?

And it happens that when you train the router you do end up passing certain classes of problem to each expert - just as a training result nothing as clean as a python expert. But math vs creative writing will tend to rely on different experts over the majority of the inference?

I do not know what I am talking about, this is my limited understanding...

Re: AirLLM 70B inference with single 4GB GPU

#45

Post by bensyverson »

parineum wrote: post #41
nickpsecurity wrote: post #30

Earlier quoted context omitted.

If I could justify wear and tear and electricity, I was willing to do something like this for batch processing. The batches would be a bunch of prompts whose outputs I'd look at the next day. Maybe common operations, like QA or refactoring, on whatever software I wrote. If so, I could use a larger model than I have real-time hardware for. The largest, well-trained models can often get the output mostly right in one t…

at 5 minutes per token, you could look at the results next week

one week later: "It says 'You're absolutely right! Let me look at the seams so I'm checking, not guessing—' and I guess that's when my SSD melted."

Re: AirLLM 70B inference with single 4GB GPU

#46

Post by esafak »

tecleandor wrote: post #39
hiramwen wrote: post #8

Earlier quoted context omitted.

You don't really need a maintainer when codex or claude code can set it up for you; thats how I got Trellis2 working on windows and tiny VRAM despite Microsoft recommending you have 24GB VRAM and Linux. Models are pretty disposable now.

So, instead of having one or two maintained projects that work super well, now we have hundreds or thousands of people half-assing it locally each time they need it? Doesn't sound efficient.

I have the same objection to forking open source projects; changes should be upstreamed when possible.

Re: AirLLM 70B inference with single 4GB GPU

#47

Post by aaroninsf »

imenani wrote: post #4

For anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token https://github.com/lyogavin/airllm/releases/tag/v3.1.0

Wow, you could do a lot at 292 tokens a sec—oh.

I have all praise for those taking this on and in my idiom would call it *the lord's work."

The image I reliably summon to mind is that compilation video showing the progress of Boston Dynamics bots. The curve between technically functional, to comically slow, to too slow for "real" work, on to, OMFG, may prove a (rough) curve.

It's work like this that moves things forward.

Re: AirLLM 70B inference with single 4GB GPU

#50

Post by IAmGraydon »

tecleandor wrote: post #39
hiramwen wrote: post #8

Earlier quoted context omitted.

You don't really need a maintainer when codex or claude code can set it up for you; thats how I got Trellis2 working on windows and tiny VRAM despite Microsoft recommending you have 24GB VRAM and Linux. Models are pretty disposable now.

So, instead of having one or two maintained projects that work super well, now we have hundreds or thousands of people half-assing it locally each time they need it? Doesn't sound efficient.

It might make sense to do this if there literally isn't an existing solution out there, but I think most would find that it's better to use an existing solution than to recreate the wheel every time there's a need. It's just less effort/time/money.
Post reply on HN