Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

neomindlabs.com

Post reply on HN
Print view View original

151–160 of 227 posts

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#151

Post by Wowfunhappy »

sigmoid10 wrote: post #125
Wowfunhappy wrote: post #124

Earlier quoted context omitted.

How fast does a human write code?

It's fast if you want to automate things that run independently or overnight. It's slow if you want to iterate code together with it.

If you were iterating on code with a human partner, how fast would they write?

Like, obviously faster is better and the entire point of computers is to do things faster, but I find it kind of surprising how many people consider 9 tokens per second—heck, even much less than that—to be unusably slow. It's still automating a class of task that virtually no one before 2022 was able to automate, and it's faster than basically any human can write code.

I guess the better question is, why would you use a 9 token per second system when you could use a much faster cloud model. Obviously, if you want speed, that's the way to go. But a lot of people seem to find the idea of sending their requests to a third party server untenable. If 9 tokens per second is the best you can do—I don't know, that seems usable and useful to me.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#152

Post by [deleted] »

dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

[deleted]

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#153

Post by [deleted] »

dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

[deleted]

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#154

Post by nmfisher »

RussianCow wrote: post #70
LeifCarrotson wrote: post #38

Earlier quoted context omitted.

I think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare! https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of lik…

Once you've used a model that runs at hundreds of TPS, it's hard to go back. Everything completes so quickly that you can iterate without breaking out of flow state. My biggest gripe with slow (<50tps) LLMs is that I've lost all the mental context I built up by the time it's done, which makes it extremely difficult to explore or iterate on solutions.

Completely agree. Slow but smart models (Fable, Sol, GLM5.2 etc) are great, but they leave me with zero mental model of the code that's been written. Most of the time my mind wanders off and I go check social media or fire off a prompt for some other random project, it's a big productivity drain.

Working with models that are super fast, but slightly dumber (like mimo-v2.5-pro-ultraspeed) is amazing, I feel like I'm still the one that's actually making every decision.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#156

Post by smeeth »

acchow wrote: post #138
smeeth wrote: post #51

Earlier quoted context omitted.

Probably won't have to wait that long. Prism released Bonsai 27B ( https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit ) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one. It is debatable if we will actually need that…

We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?

1. yes. https://www.alphaxiv.org/abs/2607.bonsai-27b table 14 shows that bonsai retains roughly 95% of the fp16 27b model's average performance and outperforms post-training quantization at a similar bit width. it doesn't directly compare against every top 10-18b model, but it is clearly still performing like a large model.

2. quantization != native low precision training. a model trained in native ternary should generally outperform a full-precision model quantized after the fact.

even if a ternary model only retains 90-95% of the performance of its fp16 equivalent, who cares? if a 200b ternary model retains most of the capability of the 200b fp16 model while using a fraction of the memory and bandwidth, it can be substantially less efficient per parameter and still dominate a smaller fp16 model under the same hardware budget.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#157

Post by Figs »

throwawayffffas wrote: post #120
hagen8 wrote: post #44

Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally. If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider. I estimate that the server consumes probably around 500W during inference. In Germany where 1kwh cost arou…

I run qwen 27b at home when working it pulls around 400W. I get 40ish tokens per second generation and more importantly about 1000 tokens per second prompt processing. In an hour it can process 3.6 million tokens or generate 144000 tokens. This costs me about 15 cents given my electricity prices. For sonnet the equivalent token costs are 7.2 dollars for the prompt processing or 1.4 dollars for the generation. The clo…

> it pulls around 400W

Try dropping the power cap on your GPU if it supports it; you can often get much lower energy usage with minimal loss of tok/s (particularly during generation) than whatever the GPU defaults to. There's a sweet spot around 200W on the GPU I'm currently testing that gives me about ~75% of the max pp and 97% of tg while using 100W less than the default/max 300W power cap -- and the card runs much quieter as a result.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#158

Post by andrekandre »

dwa3592 wrote: post #140
Aurornis wrote: post #129

Earlier quoted context omitted.

I have a free perplexity account from some promotion. Not sure what comparison you’re trying to make because Perplexity’s whole thing is that it’s really fast. It launches the search with parallel agents and then even seems to render some of the output paragraphs with parallel sessions to get the results. Doing the same thing at 7-9 tokens per second, concurrency of 1, would take ages for all of the tool calling and…

Its really easy to argue against local models because when it comes to quality, you can argue using the tokens/sec. and when it comes to speed, you can argue using the parameter count. This is not compared to the frontier stuff but it is the frontier of last year that now runs on a local machine. It was impossible to do this last year.

the local models open source harnesses are really improving quite fast; just a few months ago i couldn't get any tool calling to work, and responses were very slow (thinking going on too long etc) but now with some newer models on my macbook air, tool calling works and depending on the model, it returns from thinking fairly swiftly....

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#159

Post by walrus01 »

Capricorn2481 wrote: post #46
dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

I tried Qwen3.6-35B-A3B, but it couldn't generate a 50-100 line Clojure file without having broken parens mismatches. I know Clojure isn't super popular, but the syntax is pretty simple and the frontier models do fine with it.

what quantization? what temperature?

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#160

Post by andrekandre »

kennywinker wrote: post #139
iwontberude wrote: post #110

Earlier quoted context omitted.

Because for the company that will actually do it, their biggest customers aren’t data centers they are iPhone owners.

First off the math doesn’t math. Datacenters are willing to pay $50k for a single high end GPU. If you have unlimited capacity, yeah sell millions for $100 a pop or $10 a pop or whatever the bom cost of a phone GPU would be - but if you have limited capacity, you’re gonna sell all of that to the customer who is willing to pay the most PER UNIT. Second off, this doesn’t work from a power consumption standpoint. When I…

  > Datacenters are willing to pay $50k for a single high end GPU.
its true for now, because capital is flowing like a torrent, but how long will that last if returns start to be expected (aka the bubble pops)?
Post reply on HN