That's quite slow I'm getting 8-12 t/s on a 13 year old CPU. (Speed varies by context size and other settings who knows) https://news.ycombinator.com/item?id=48354801
Thank you for sharing / linking!
Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
11–20 of 227 posts
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#12Here's my report running several different models on a dual Xeon with 256 GB of DDR4 and no GPU. https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5...
Thank you for sharing!
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#13Here's my report running several different models on a dual Xeon with 256 GB of DDR4 and no GPU. https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5...
Have you tried with a single CPU to get rid of the NUMA penalty? I understand this likely means halving the memory but I am interested in how much of a difference it makes
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#14Is it just me or does this post not mention how much RAM they had? I would love to know - I have a dual-Xeon 1U screamer with 96GB of DDR4 RDIMM just sitting around...
FWIW I'm getting a hardware max of 20 tok/s (approx topping out the GPU's compute) on my custom local diffusiongemma port running on an M3.
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#15Author here. The short version: a viral post ran Gemma 4 on a 2016 Xeon; my Xeons are 2013, and the fork it used assumes AVX2, which Ivy Bridge doesn't have. The build failure was easy. The fun bug was the silent one: two MoE graph ops with no dispatch case on non-AVX2 builds, so every expert FFN output was uninitialized memory. Deterministic, NaN-free, fluent-looking multilingual gibberish. The fix is open upstream…
This reads as pretty clearly AI-generated text, which is against HN guidelines.
The PR? He said it was AI in the comment you replied to...
I don't think the post itself reads like AI at all, but that's just me.
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#16Here's my report running several different models on a dual Xeon with 256 GB of DDR4 and no GPU. https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5...
Have you tried with a single CPU to get rid of the NUMA penalty? I understand this likely means halving the memory but I am interested in how much of a difference it makes
I have (192GB machine with two CPUs), pretty much does the trick. It just runs some small models used for embedding, etc. and has those on one CPU / memory node and all the Docker containers on the other one.c
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#17Is it just me or does this post not mention how much RAM they had? I would love to know - I have a dual-Xeon 1U screamer with 96GB of DDR4 RDIMM just sitting around... FWIW I'm getting a hardware max of 20 tok/s (approx topping out the GPU's compute) on my custom local diffusiongemma port running on an M3.
hey, I’m the author. That box has 384gb, but loading the model “only” uses about 80gb.
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#18Need to run this on my Xeons with AMX
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#19He's shown me his set up in his basement. It's sick! Talk about your 3d printer next!
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#20To me context means everything.
Tokens per second is a great metric but in the real world context window is the deal breaker when a real use case is on the table.