Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

neomindlabs.com

Post reply on HN
Print view View original

191–200 of 227 posts

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#191

Post by throwawayffffas »

why_only_15 wrote: post #167
throwawayffffas wrote: post #120

Earlier quoted context omitted.

I run qwen 27b at home when working it pulls around 400W. I get 40ish tokens per second generation and more importantly about 1000 tokens per second prompt processing. In an hour it can process 3.6 million tokens or generate 144000 tokens. This costs me about 15 cents given my electricity prices. For sonnet the equivalent token costs are 7.2 dollars for the prompt processing or 1.4 dollars for the generation. The clo…

The correct comparison is not sonnet, but qwen3.5-27b on a cloud. Alibaba's pricing [0] is $0.20/m input $1.56/m output, so $0.72 for the prompt processing or $0.22 for the generation. Yours is still cheaper but the margins are less. My guess is that this math gets less good with MoE (because you will be limited by VRAM, but clouds won't). [0]: https://openrouter.ai/qwen/qwen3.5-27b

I run 3.6 but yeah you are right

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#192

Post by puzzlingcaptcha »

ryandrake wrote: post #36
trollbridge wrote: post #16

Earlier quoted context omitted.

I have (192GB machine with two CPUs), pretty much does the trick. It just runs some small models used for embedding, etc. and has those on one CPU / memory node and all the Docker containers on the other one.c

I have a dual xeon also, same as OP: Ivy Bridge + 128GB DRAM, and was never really able to get decent LLM performance out of it. So I ended up biting the bullet and adding a "budget tier" A4000 20GB GPU. Too bad all my DRAM is wasted now--not sure if there is a way to take advantage of lots of DRAM once you move over to having inference happening on the GPU.

Have you tried putting the KV cache on the GPU and running inference from RAM? From what I gather, prompt processing is particularly painful using RAM alone.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#193

Post by sigmoid10 »

Wowfunhappy wrote: post #151
sigmoid10 wrote: post #125

Earlier quoted context omitted.

It's fast if you want to automate things that run independently or overnight. It's slow if you want to iterate code together with it.

If you were iterating on code with a human partner, how fast would they write? Like, obviously faster is better and the entire point of computers is to do things faster, but I find it kind of surprising how many people consider 9 tokens per second—heck, even much less than that—to be unusably slow. It's still automating a class of task that virtually no one before 2022 was able to automate, and it's faster than basic…

I guess you haven't used Codex/Copilot much? All good LLMs now use reasoning. Especially for code it is almost a requirement to produce correct results. But that means even having them implement a comparatively simple function or algorithm can easily produce 3000 output tokens. Waiting 5+ minutes every time you iterate because you want a tiny tweak in the code is not a practical strategy for live development.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#195

Post by fluoridation »

pmarreck wrote: post #165
allknowingfrog wrote: post #135

Earlier quoted context omitted.

There's a whole spectrum of employment between faceless corporations and personal side projects. AI will replace humans because giant business believe they can do the same work, not because they will actually be able to. The correctness of an application is limited by your ability to understand and describe what you need. We have a word for an application specification tool so detailed it eliminates all ambiguity. It…

> AI will replace humans because giant business believe they can do the same work, not because they will actually be able to. This has been being claimed for at least 2 years now. Wouldn't we already be seeing disasters if this was the case? It's certainly been around long enough to cause some real damage. Instead there is a slow trickle of things that makes the news by people who didn't bother instituting a single f…

>If you write code in Lean 4 or Idris 2, you may not completely understand why it is or isn't correct, but their respective compilers will certainly prove it to you one way or the other.

No, the prover can only prove that the implementation matches the formal language specification. That's different from the application being correct.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#196

Post by mseri »

mseri wrote: post #183
dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

Nice! I had done the exact same with gemma4 26b, both for my Intel laptop and for my M1 with 8Gb RAM (with also q4 and turboquant). I don’t use it much since there are dumber but way faster models to run, but I should clean up the code and make it available

Ok, done: https://github.com/mseri/zunzuncito My main focus is systems with very low ram. On my M1 with 8Gb or RAM llama.cpp cannot run gemma4 26b, but this implementation works fine with 5-6 tok/s.

I also replaced my crap implementation of the oai server following yours, I hope it's all right. I did add a mention to your project and this fact in the README. It is a lot more barebones than what you have, but I have to admit that it works really nicely for me

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#198

Post by imtringued »

Wowfunhappy wrote: post #151
sigmoid10 wrote: post #125

Earlier quoted context omitted.

It's fast if you want to automate things that run independently or overnight. It's slow if you want to iterate code together with it.

If you were iterating on code with a human partner, how fast would they write? Like, obviously faster is better and the entire point of computers is to do things faster, but I find it kind of surprising how many people consider 9 tokens per second—heck, even much less than that—to be unusably slow. It's still automating a class of task that virtually no one before 2022 was able to automate, and it's faster than basic…

You have to remember that the CoT thinking tokens take a while to produce. If your model produces 900 thinking tokens you're waiting almost two minutes for the first token.

The biggest argument in favor of running local models doesn't seem to be privacy at all, it's the fact that you can't run out of tokens even if it is a bit slow.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#199

Post by imtringued »

Aurornis wrote: post #75
dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

> I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. This prediction alone isn’t useful at all without a bound on speed and maybe quantization. You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because the…

>If you’re saying that hardware will catch up by mid-2027, I disagree.

If DDR6 comes out that will actually double the effective memory bandwidth on next generation computers. Most computers will reach a good fraction of a Strix Halo system's memory bandwidth and the next generation of Strix Halo will reach Macbook Pro levels of memory bandwidth. Of course there is no guarantee this will happen by 2027 but DDR6 will probably exist in some form by the end of 2027.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#200

Post by [deleted] »

dwa3592 wrote: post #122
felineflock wrote: post #117

Earlier quoted context omitted.

That is awesome! I am curious about the decision to not use GPU since this is for Apple Silicon. Wouldn't the GPU potentially accelerate the DeltaNet/attention layers and matrix multiplication in general?

It will, but the process at this point is SSD bound rather than compute bound. On a bigger machine, Apple silicon must help but I don't have a bigger machine. I can think about this more and will make changes if that helps.

[deleted]
Post reply on HN