Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
1–10 of 227 posts
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#2Author here. The short version: a viral post ran Gemma 4 on a 2016 Xeon; my Xeons are 2013, and the fork it used assumes AVX2, which Ivy Bridge doesn't have. The build failure was easy. The fun bug was the silent one: two MoE graph ops with no dispatch case on non-AVX2 builds, so every expert FFN output was uninitialized memory. Deterministic, NaN-free, fluent-looking multilingual gibberish.
The fix is open upstream as PR #2138 (https://github.com/ikawrakow/ik_llama.cpp/pull/2138), awaiting review. Fair warning on the AI angle: the patch was written by Claude at my direction. The post is explicit about which parts were me and which weren't. Happy to answer questions about either the bug or the workflow.
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#3Author here, it looks like my original comment was flagged for some reason. The fix is open upstream as PR #2138 (https://github.com/ikawrakow/ik_llama.cpp/pull/2138)
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#4Truly amazing. This gives a peek into the future for what's possible.
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#5That's quite slow I'm getting 8-12 t/s on a 13 year old CPU. (Speed varies by context size and other settings who knows)
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#6Author here. The short version: a viral post ran Gemma 4 on a 2016 Xeon; my Xeons are 2013, and the fork it used assumes AVX2, which Ivy Bridge doesn't have. The build failure was easy. The fun bug was the silent one: two MoE graph ops with no dispatch case on non-AVX2 builds, so every expert FFN output was uninitialized memory. Deterministic, NaN-free, fluent-looking multilingual gibberish. The fix is open upstream…
This reads as pretty clearly AI-generated text, which is against HN guidelines.
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#7Here's my report running several different models on a dual Xeon with 256 GB of DDR4 and no GPU.
https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5...
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#8Related:
A 10 year old Xeon is all you need
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#9Author here. The short version: a viral post ran Gemma 4 on a 2016 Xeon; my Xeons are 2013, and the fork it used assumes AVX2, which Ivy Bridge doesn't have. The build failure was easy. The fun bug was the silent one: two MoE graph ops with no dispatch case on non-AVX2 builds, so every expert FFN output was uninitialized memory. Deterministic, NaN-free, fluent-looking multilingual gibberish. The fix is open upstream…
This reads as pretty clearly AI-generated text, which is against HN guidelines.
Here's the thing: life also imitates art. If you invert your load-bearing assumption, it could be that he just reads too much slop. But my honest take? You might be right.
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#10Apologies for asking here but literally nobody knows:
Android studio connected to a local model disconnects automatically after 10 minutes. How set this limit to 12 hours or remove it completely?
I could run my LM studio model all night... but I cant, since Android studio times out after a hard limit of 10M.
This is not related to number of tokens.
I tried Googling, searching for settings in Android studio, even created a stackoverflow post - but zero information. Jetbrains mentions "remote agent timeout mechanism" - but after changing it, nothing happens.