Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

neomindlabs.com

Post reply on HN
Print view View original

141–150 of 227 posts

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#141

Post by giantrobot »

fouc wrote: post #33
neomindryan wrote: post #17

Earlier quoted context omitted.

hey, I’m the author. That box has 384gb, but loading the model “only” uses about 80gb.

any reason you went with q8 over q4? I'm wondering if q4 would run noticeably faster or not.

Token generation is nominally bandwidth limited. Prefill/prompt processing is nominally compute limited.

For CPU inference on old hardware I don't think q4 offers any benefit over q8 since the AVX unit doesn't support such small floats. I don't even think AVX supports 4-bit int math. IIRC AVX2 does.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#142

Post by flockonus »

smeeth wrote: post #51
dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

Probably won't have to wait that long. Prism released Bonsai 27B ( https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit ) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one. It is debatable if we will actually need that…

There is also the 1bit version @ 3.9 GB that retains 90% of the intelligence - quite a feat!

https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#143

Post by kQq9oHeAz6wLLS »

throwaway2027 wrote: post #5

That's quite slow I'm getting 8-12 t/s on a 13 year old CPU. (Speed varies by context size and other settings who knows) https://news.ycombinator.com/item?id=48354801

Yeah, I'm seeing 8-9 t/s on a Xeon CPU E3-1270 V2 @ 3.50GHz with an old Nvidia Quadro K2200 (4GB). I run gemma4:e2b and gemma4:12b-it-qat on Ollama.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#144

Post by razster »

dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

I sense the same. Running Ornith 35b with Pi and hitting 50+toks, and now that I learned Pi can do search and fetch, I have not needed to visit any of the large models as a search function. Soon we will be seeing new models drop this month and next that will change some of the landscape. Exciting times. P.s. Try Ornith, well worth it.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#145

Post by cyanydeez »

razster wrote: post #144
dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

I sense the same. Running Ornith 35b with Pi and hitting 50+toks, and now that I learned Pi can do search and fetch, I have not needed to visit any of the large models as a search function. Soon we will be seeing new models drop this month and next that will change some of the landscape. Exciting times. P.s. Try Ornith, well worth it.

setup deer-flow: https://github.com/bytedance/deer-flow

its pretty good for building any kind of research report using search engines and it manages context corrextly.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#146

Post by nkozyra »

acchow wrote: post #138
smeeth wrote: post #51

Earlier quoted context omitted.

Probably won't have to wait that long. Prism released Bonsai 27B ( https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit ) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one. It is debatable if we will actually need that…

We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?

No.

But do you need to run every small problem through a 10B-30B model?

We're smashing ants with hammers most of the time. We're asking frontier Opus/Fable models to classify text and build frontend code.

Once we start dissecting these problems into smaller discreet tasks and having the big reasoning models do the tough stuff, we suddenly have an economical system. Not for the company hoping for a big IPO, but for the end user.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#147

Post by yieldcrv »

acchow wrote: post #138
smeeth wrote: post #51

Earlier quoted context omitted.

Probably won't have to wait that long. Prism released Bonsai 27B ( https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit ) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one. It is debatable if we will actually need that…

We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?

Its somewhat good, the prism team’s webgpu demo gave it a couple dozen “kernels” written in Fable 5 and it calls them for almost everything procedural

I feel like these things are experiencing convergent evolution to be like biological brains. The large parameters are merely potentially large parameters and they keep having more and more and smaller active layers, which are themselves quantized down. This is seems analogous to the chemical spiking of neurons and inactive layers of a brain in power and efficiency.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#148

Post by dTal »

dwa3592 wrote: post #54
smeeth wrote: post #51

Earlier quoted context omitted.

Probably won't have to wait that long. Prism released Bonsai 27B ( https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit ) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one. It is debatable if we will actually need that…

agreed!! in my heart i really wanted to say by the end of 2026 but wanted to add some wiggle room in case they start to ban open source AI development.

>in case they start to ban open source AI development

Is that likely, do you think?

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#149

Post by RussianCow »

hx8 wrote: post #136
RussianCow wrote: post #70

Earlier quoted context omitted.

Once you've used a model that runs at hundreds of TPS, it's hard to go back. Everything completes so quickly that you can iterate without breaking out of flow state. My biggest gripe with slow (<50tps) LLMs is that I've lost all the mental context I built up by the time it's done, which makes it extremely difficult to explore or iterate on solutions.

I'd rather have slower and better output than worse and faster output.

It depends. For something high stakes or inherently complex, sure, you don't want to have to clean up the agent's mess afterwards. But for many tasks like building web UIs, the difference in output quality is going to be small enough that iteration speed will win over quality.

With a fast enough model, I can iterate on the UI of a given screen 4-5 times before Opus finishes its first attempt.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#150

Post by dwa3592 »

dTal wrote: post #148
dwa3592 wrote: post #54

Earlier quoted context omitted.

agreed!! in my heart i really wanted to say by the end of 2026 but wanted to add some wiggle room in case they start to ban open source AI development.

>in case they start to ban open source AI development Is that likely, do you think?

dario has been saying open source models are dangerous. who knows who is listening to him.
Post reply on HN