Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

neomindlabs.com

Post reply on HN
Print view View original

201–210 of 227 posts

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#201

Post by ubercore »

nkozyra wrote: post #146
acchow wrote: post #138

Earlier quoted context omitted.

We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?

No. But do you need to run every small problem through a 10B-30B model? We're smashing ants with hammers most of the time. We're asking frontier Opus/Fable models to classify text and build frontend code. Once we start dissecting these problems into smaller discreet tasks and having the big reasoning models do the tough stuff, we suddenly have an economical system. Not for the company hoping for a big IPO, but for th…

The big hammers buy you more confidence and need less supervision. In pure task execution you _might_ smash the ant with a small surgical hammer, but if you absolutely need it smashed, that's when people still reach for the big hammer. It buys more confidence.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#203

Post by zozbot234 »

Aurornis wrote: post #75
dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

> I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. This prediction alone isn’t useful at all without a bound on speed and maybe quantization. You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because the…

> You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.

On the contrary, that's ultimately an excellent baseline capability for most casual LLM users. Especially if you add the ability to fire off multiple requests over time and work on them concurrently (at least for not-very-long contexts), which is quite natural if one can implement some sort of continuous batching.

Even quantizing the model (with the ensuing loss in quality) is not an absolute requirement, quite unlike e.g. ensuring that a local model can run fully from VRAM on a typical consumer dGPU.

In general, ISTM that people overemphasize real-time or near real-time response which is a rather terrible fit if your goal is efficient LLM inference of near-SOTA models on typical local hardware.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#204

Post by benjiro29 »

smeeth wrote: post #156
acchow wrote: post #138

Earlier quoted context omitted.

We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?

1. yes. https://www.alphaxiv.org/abs/2607.bonsai-27b table 14 shows that bonsai retains roughly 95% of the fp16 27b model's average performance and outperforms post-training quantization at a similar bit width. it doesn't directly compare against every top 10-18b model, but it is clearly still performing like a large model. 2. quantization != native low precision training. a model trained in native ternary should gen…

Models are not like a fps counter where losing a few percentage has no actual impact. These percentages may be the difference between a model that writes code and one that goes into loops, and think rm -rf . is a good idea.

There is a reason why most models try to stay in the FP4 or higher range, because the reduced accuracy can have major consequences.

You are better off with a 8b FP4+ model then a 27b Q2 model.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#205

Post by 1d22a »

hagen8 wrote: post #44

Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally. If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider. I estimate that the server consumes probably around 500W during inference. In Germany where 1kwh cost arou…

To this end, I've been thinking that it would be cool to create a solar-battery powered ("off grid") server providing a self-hosted LLM service. Offline when the sun doesn't shine enough (like Low Tech Magazine[1]). At whatever size is required for every day, community-scale use (a friend group, a street, a club). Fix the data centre issue by democratising AI to such an extent that we can bring it into the hands of communities to actually control (and democratically decide their own level of censorship / alignment). Along the lines of some of Geohotz' writing.

Within a short time I think open source models will all be getting good and efficient enough to make it viable to serve this on 2nd hand hardware for cheap. All it will take is a nerd in every small community to pool together a few hundred bucks initial outlay, and then ongoing costs are near free without electricity to pay for.

[1]: https://solar.lowtechmagazine.com/

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#206

Post by javier2 »

dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

How do I install without curl-piping to bash?

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#207

Post by damienmeur »

dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

Just wake me u when I can run fable or gpt 5.6 for free my raspberry-pi 4 at 2000 tokens/s

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#208

Post by broabprobe »

sjs382 wrote: post #119
broabprobe wrote: post #26

I run the same setup Gemma 4 26B on a 2013 Mac Pro (dual graphics cards but they're useless for this). I also get about 5 t/s. It's perfectly serviceable for some tasks!

I bought a trashcan Mac Pro on a whim last week ($120 in eBay!) and did some reading about them—it turns out people recently started using the GPUs to run models @ 20-30 tok/s. I'm excited to get my mitts on it on Friday when it finally arrives. Here's some of the resources I came across if you're interested in reading. https://echalupa.com/blog/mac-pro-6-1-llama-cpp-firepro-d300... https://matthewgribben.com/blog/ma…

oh very exciting! Thanks for sharing these sjs382! A bummer the models can't be run on both the GPUs and CPU so you could run much larger models. A year ago I was running gpt120b as it easily fit in 128gb ram. But now I'm only running gemma 26b and wishing I had stuck with 64gb ram since 128gb gets throttled. Not regretting buying 128gb 1.5 years ago though!

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#209

Post by minkeymaniac »

dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

I just downloaded llama. ran this llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

I am getting [ Prompt: 91.3 t/s | Generation: 171.8 t/s ]

This is on a GPU (RTX 4060)

Is this decent?

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#210

Post by pjmlp »

pmarreck wrote: post #165
allknowingfrog wrote: post #135

Earlier quoted context omitted.

There's a whole spectrum of employment between faceless corporations and personal side projects. AI will replace humans because giant business believe they can do the same work, not because they will actually be able to. The correctness of an application is limited by your ability to understand and describe what you need. We have a word for an application specification tool so detailed it eliminates all ambiguity. It…

> AI will replace humans because giant business believe they can do the same work, not because they will actually be able to. This has been being claimed for at least 2 years now. Wouldn't we already be seeing disasters if this was the case? It's certainly been around long enough to cause some real damage. Instead there is a slow trickle of things that makes the news by people who didn't bother instituting a single f…

I can easily assert than in corporation consulting there are no human translation teams any longer, all translations are AI driven, and at very least there might exist some QA regarding those translations.

Likewise for image assets for CMS and ecommerce sites, now instead of hiring agencies, many of those images are AI generated.

Just because it hasn't come yet to some coder jobs, doesn't mean others haven't been already displaced by it.

Post reply on HN