I run the same setup Gemma 4 26B on a 2013 Mac Pro (dual graphics cards but they're useless for this). I also get about 5 t/s. It's perfectly serviceable for some tasks!
What is it useful for at slow speed?
Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
71–80 of 227 posts
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#72I was inspired by that post also, got a Qwen Coder 1.5B up to 27tok/s prompt eval and 13tok/s decode on an e5-2650v2 inside a GNOME box
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#73Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally. If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider. I estimate that the server consumes probably around 500W during inference. In Germany where 1kwh cost arou…
Locally I’m looking at about CAD $0.05 per kWh when off peak.
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#74Good job. Thanks for sharing. I have a similar NAS server with 2 Intel(R) Xeon(R) CPU E5-2699 CPUs. I will test as well.
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#75I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
> I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware.
This prediction alone isn’t useful at all without a bound on speed and maybe quantization.
You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.
If you’re saying that hardware will catch up by mid-2027, I disagree. The limitation is fast memory and that’s going to be expensive for a while. I have a 128GB unified memory machine that can technically run 200B MoE models with enough quantization, but it’s so slow that there’s no reason to do it. It’s going to be a few years before we have enough RAM and processing power in basic consumer hardware without spending as much as a used car to get it.
> This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
Sorry, but 9 tokens per second with a slow prompt processing speed is not decent for anything other than getting short chat responses.
You’re also not running the full GPT4 quality model. I’m very familiar with that model from some other work and the 4-bit quants are just not as good as all of those KL divergence plots would have you believe.
You also have very short context. It’s basically useless for anything more than short chats where you’re okay watching the output come back at reading speed, skipping the reasoning part (which is important for calling it GPT4 level quality), and waiting a long time for the first token.
Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#76Apologies for asking here but literally nobody knows: Android studio connected to a local model disconnects automatically after 10 minutes. How set this limit to 12 hours or remove it completely? I could run my LM studio model all night... but I cant, since Android studio times out after a hard limit of 10M. This is not related to number of tokens. I tried Googling, searching for settings in Android studio, even crea…
If the local model is served via ollama, there's a default timeout of 10 minutes , which can be adjusted either per-call , or (as I did) in the systemd service environment variables https://docs.ollama.com/faq#how-do-i-keep-a-model-loaded-in-... You didn't specify what was serving your local model.
Thank you for your reply. I use LM studio (local server), but can switch to a different tool.
Do you know how to switch it in LM studio?
What I see is that: android studio gives "Error: stream failed" and in LM studio server I see it is still working, then says that client (=android studio) disconnected.
So I assumed it was a setting in android studio.
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#77Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally. If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider. I estimate that the server consumes probably around 500W during inference. In Germany where 1kwh cost arou…
It would be more efficient if you had multiple users (or agents) making parallel requests to take advantage of batching, right?
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#78Earlier quoted context omitted.
> on a decent speed But you said 7-9 tokens/second , that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.
I think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare! https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of lik…
> Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete.
If I spend 10 minutes reading an article, that would only generate 3000 tokens.
That’s not counting the prompt processing time.
We have very different expectations for LLMs if your tasks only take a couple thousand tokens and you’re happy waiting 10 minutes for it.
> Yes, with top-tier GPU farms you can hit hundreds of tokens per second
My 5090 gets hundreds of tokens per second with this model. No farm needed. I’d have to double check but I think even a $1000 Intel B70 might break 100 tokens per second.
> But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
If that old Xeon pulls 200W from the wall and you pay national average electricity costs, it’s going to cost $0.90 per day to run it.
I would rather pay a dollar per day, get my answers 100X faster, and not have an old Xeon heating up my house.
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#79Earlier quoted context omitted.
i am working on making it faster but to me 7-9 tokens/sec feels very good. it was 0 tokens/sec a year ago.
Ignore the haters. What you've done is incredible!
If you’re interested in these projects you should check out the project this was based on: https://github.com/JustVugg/colibri
It says so right in the readme. They’re not hiding anything.
Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
#80Earlier quoted context omitted.
The PR? He said it was AI in the comment you replied to... I don't think the post itself reads like AI at all, but that's just me.
I think "this" refers to its parent comment. Part of it sounds like Claude wrote it. AI-generated comments aren't allowed on HN.
Indeed, I was referring to the parent comment.