Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

neomindlabs.com

Post reply on HN
Print view View original

81–90 of 227 posts

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#81

Post by Aurornis »

A dual Xeon of this era is probably pulling 300W or more when loaded.

At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space.

If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000 tokens in a day.

That’s about $0.30 per million output tokens.

Coincidentally, that’s the same price for this model on OpenRouter right now, but OpenRouter token gen will be 8X faster.

There are a lot of good reasons to experiment with running LLMs locally, like if you don’t want any data leaving your house.

Don’t think that you’re going to come out ahead monetarily. I say this as someone with a lot more money invested in local inference hardware at home. It’s fun, but it’s not a way to save money.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#82

Post by brailsafe »

Aurornis wrote: post #81

A dual Xeon of this era is probably pulling 300W or more when loaded. At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space. If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000 tokens in a day. That’s about $0.30 per million output tokens. Coincidentally, that’s the same price for this model on Ope…

Reasonable analysis, especially because this person seems to have an actual house. In my case, I rent and don't pay for electricity directly, so the cost effectiveness threshold is whenever the landlord starts complaining

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#83

Post by twsted »

Aurornis wrote: post #81

A dual Xeon of this era is probably pulling 300W or more when loaded. At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space. If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000 tokens in a day. That’s about $0.30 per million output tokens. Coincidentally, that’s the same price for this model on Ope…

I think, may be actually wrong, that most of us do not consider running a model locally a way to save money. It is a way not to spread personal info around.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#84

Post by Capricorn2481 »

Azantys wrote: post #59
Capricorn2481 wrote: post #46

Earlier quoted context omitted.

I tried Qwen3.6-35B-A3B, but it couldn't generate a 50-100 line Clojure file without having broken parens mismatches. I know Clojure isn't super popular, but the syntax is pretty simple and the frontier models do fine with it.

You are comparing a 35B models to a 635B+ frontier model, of course thats not even close

I'm not lamenting that they aren't close, I'm saying Qwen will frequently output code that isn't even syntactically correct, even when the syntax is simple. Which makes it unusable for coding.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#86

Post by Aurornis »

twsted wrote: post #83
Aurornis wrote: post #81

A dual Xeon of this era is probably pulling 300W or more when loaded. At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space. If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000 tokens in a day. That’s about $0.30 per million output tokens. Coincidentally, that’s the same price for this model on Ope…

I think, may be actually wrong, that most of us do not consider running a model locally a way to save money. It is a way not to spread personal info around.

Anyone running LLMs at home will come to that realization quickly, if they’re looking at their power bills. Even feeling the heat output of a computer running at 100% in your office makes it clear.

I was responding to a lot of the comments saying this was a reasonable way to avoid paying for tokens or subscriptions. I don’t want anyone getting the wrong idea that this is a way to save money if that’s their priority.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#87

Post by zoobab »

Aurornis wrote: post #78
LeifCarrotson wrote: post #38

Earlier quoted context omitted.

I think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare! https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of lik…

> Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. If I spend 10 minutes reading an article, that would only generate 3000 tokens. That’s not counting the prompt processing time. We have very different expectations for LLMs if your tasks only take a couple thousand tokens and you’re happy waiting 10 minutes for it. > Yes, with top-tier GPU farms you can hit hundreds…

You could suspend it to ram, and only wake it up on request, it takes 2 seconds on my box.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#88

Post by allknowingfrog »

nolok wrote: post #68
allknowingfrog wrote: post #60

Earlier quoted context omitted.

We clearly have different goals. I want an LLM to review my code, not the other way around.

It's still the same thing, you can ask it to do a full on report give explanation and details be thorough and then go do something else, another task a lunch break whatever and it will be done when you're back

How do you maintain a flow state during a lunch break? I'm looping with Claude on a scale of minutes. While you're waiting, I'm iterating.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#89

Post by typon »

dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

By early 2028, major players like Intel, AMD, QC will ship accelerators in consumer laptops capable of running ~1T MoE models at ~100 tok/s

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#90

Post by IsTom »

Capricorn2481 wrote: post #84
Azantys wrote: post #59

Earlier quoted context omitted.

You are comparing a 35B models to a 635B+ frontier model, of course thats not even close

I'm not lamenting that they aren't close, I'm saying Qwen will frequently output code that isn't even syntactically correct, even when the syntax is simple. Which makes it unusable for coding.

To be fair, they don't have the text editor highlighting all the matching parens. I'd be lost too.
Post reply on HN