Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

neomindlabs.com

Post reply on HN
Print view View original

181–190 of 227 posts

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#181

Post by parineum »

LeifCarrotson wrote: post #38
reidrac wrote: post #27

Earlier quoted context omitted.

> on a decent speed But you said 7-9 tokens/second , that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.

I think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare! https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of lik…

> there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.

That Xeon is going to be devouring power.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#182

Post by senordevnyc »

albrewer wrote: post #62
hagen8 wrote: post #44

Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally. If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider. I estimate that the server consumes probably around 500W during inference. In Germany where 1kwh cost arou…

Not OP, but your math is a bit off - I have solar panels :)

There is something genuinely beautiful to me that these goofy apes have managed to turn sunlight and sand into intelligence.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#183

Post by mseri »

dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

Nice!

I had done the exact same with gemma4 26b, both for my Intel laptop and for my M1 with 8Gb RAM (with also q4 and turboquant). I don’t use it much since there are dumber but way faster models to run, but I should clean up the code and make it available

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#184

Post by leobg »

hagen8 wrote: post #44

Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally. If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider. I estimate that the server consumes probably around 500W during inference. In Germany where 1kwh cost arou…

I’m in DE, too. Same calculus. Excluding winter though. When it’s cold outside, a GPU heats my office and I get tokens free on the side. :)

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#185

Post by Loocid »

euio757 wrote: post #100
Aurornis wrote: post #86

Earlier quoted context omitted.

Anyone running LLMs at home will come to that realization quickly, if they’re looking at their power bills. Even feeling the heat output of a computer running at 100% in your office makes it clear. I was responding to a lot of the comments saying this was a reasonable way to avoid paying for tokens or subscriptions. I don’t want anyone getting the wrong idea that this is a way to save money if that’s their priority.

> Even feeling the heat output of a computer running at 100% in your office makes it clear. What does it make clear? That I can replace the space heater my wife runs 9 out of 12 months of the year with a home server? And effectively get $0.00 per token during those times? In houses running A/C year round, sure there'd be some impact, but in all the places running heat, doesn't seem that it'd move the needle on power…

There are bunch of bitcoin miner products now that are packaged to look like regular space heaters.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#186

Post by dlcarrier »

hagen8 wrote: post #44

Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally. If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider. I estimate that the server consumes probably around 500W during inference. In Germany where 1kwh cost arou…

From what I've seen, most inference providers are running at a loss, so it wouldn't be at all surprising if using their services costs less that running the same software locally.

The commodification of the hardware needed is probably a larger factor, because by the time a baseline computer has enough RAM and processing power to run a desired LLM, that hardware will be efficient enough that the extra electricity usage is nominal.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#187

Post by aaulia »

reidrac wrote: post #27
dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

> on a decent speed But you said 7-9 tokens/second , that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.

I tried Qwen MoE a while back. Using my 8GB RX470, somehow got 10+ token/sec, lot's of trial and error with llama.cpp config, and it's still slow to be used for my usecase. Even at 12 to 16 IMO it's slow. For chat, maybe it's enough, but for any other tasks, it's not viable IMO

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#188

Post by kennywinker »

typon wrote: post #177
kennywinker wrote: post #139

Earlier quoted context omitted.

First off the math doesn’t math. Datacenters are willing to pay $50k for a single high end GPU. If you have unlimited capacity, yeah sell millions for $100 a pop or $10 a pop or whatever the bom cost of a phone GPU would be - but if you have limited capacity, you’re gonna sell all of that to the customer who is willing to pay the most PER UNIT. Second off, this doesn’t work from a power consumption standpoint. When I…

There is ton of room for improvement "down there". * Software inference optimizations * Heavy quantization * Chips with hardcoded transformer architecture * Much cheaper HBM * Much sparser models - 1T total with ~1-10B active params e.g. * Not to mention - 2 years of today's frontier models writing RTL and kernels at superhuman levels.

> * Software inference optimizations

Absolutely. I'd be surprised if they couldn't 2x performance in the next year. Still doesn't make a 1T model fit on your phone.

> * Heavy quantization

I think this is a dead end if you're trying to fit a 1T model into a phone. Makes much more sense to train a model that's designed to be small, than train a model that's smart and then quantize it into stupidity.

> * Chips with hardcoded transformer architecture

Totally, this will probably work great. Now good luck booking fab time any time in the next 2 years.

> * Much cheaper HBM

Totally, this will probably work great. Now good luck booking fab time any time in the next two years.

> * Much sparser models - 1T total with ~1-10B active params e.g.

Fewer active params helps with the speed of token generation, but if the whole model doesn't fit into ram it doesn't solve the issue of having to constantly stream portions of the model from disk to ram.

> * Not to mention - 2 years of today's frontier models writing RTL and kernels at superhuman levels.

IMO this is a delusional myth-making idea being sold to us by ai companies. Machines that generate output based on statistical averages won't generate genuinely new ideas. They can help us try out ideas faster, but they're simply not capable of the kind of creativity and understanding required to push a field forward, except incrementally.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#189

Post by mdp2021 »

nkozyra wrote: post #146
acchow wrote: post #138

Earlier quoted context omitted.

We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?

No. But do you need to run every small problem through a 10B-30B model? We're smashing ants with hammers most of the time. We're asking frontier Opus/Fable models to classify text and build frontend code. Once we start dissecting these problems into smaller discreet tasks and having the big reasoning models do the tough stuff, we suddenly have an economical system. Not for the company hoping for a big IPO, but for th…

> do you need to run every small problem through a 10B-30B model? ... We're asking frontier Opus/Fable models to classify text

Actually probably yes: text analysis (magazine articles) by LLMs in the ~30b .. ~120b range failed miserably (and also randomly - the rare cases of proper interpretation occurred among the failure cases) with the main public models of around one year ago, tried extensively.

So, yes, you can employ an ~80IQ only if you will expect the related quality.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#190

Post by mdp2021 »

dwa3592 wrote: post #150
dTal wrote: post #148

Earlier quoted context omitted.

>in case they start to ban open source AI development Is that likely, do you think?

dario has been saying open source models are dangerous. who knows who is listening to him.

The one time in which I saw Juergen Schmidhuber in perfect nervous control, "coolness" they may say westward, was when he replied to one member of the audience, "The same observation was made when they invented fire: oh, it's dangerous. But in the end, now it's here (shrugh)".

There is a proposal in the USA to restrict LLM access. This will only have us depend more and more on open source models and their providers. And cause a drain of research in those areas in which it will be impeded.

Post reply on HN