Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

neomindlabs.com

Post reply on HN
Print view View original

131–140 of 227 posts

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#131

Post by dofm »

LeifCarrotson wrote: post #38
reidrac wrote: post #27

Earlier quoted context omitted.

> on a decent speed But you said 7-9 tokens/second , that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.

I think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare! https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of lik…

> It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done.

This is how I used to think about my 3D printer, but FWIW the way my actual thinking and planning works, print speed really matters. Not for the final print, but for iterative work and test parts, it is obvious that either having a fast printer helps. Having multiple slow printers also helps, but there are only so many areas of a design you can iterate on at once.

At the moment my own LLM use is experimental and iterative, and I definitely favour the faster MoE models for much of what I am doing, even if I might in principle prefer to get the final work done in the slower ones.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#132

Post by ClikeX »

fwip wrote: post #126
allknowingfrog wrote: post #88

Earlier quoted context omitted.

How do you maintain a flow state during a lunch break? I'm looping with Claude on a scale of minutes. While you're waiting, I'm iterating.

You don't "maintain flow." You eat lunch. I swear, tech culture has gotten people wanting to work for the machines, rather than the other way round.

Right? Tech should make my work easier. Not have me stressing out even more.

Let the machine do its work while I relax, I’ll check up on it later.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#133

Post by TacticalCoder »

ChrisArchitect wrote: post #8

Related: A 10 year old Xeon is all you need https://news.ycombinator.com/item?id=48353348

Yes and a 10 year old Xeon is going to be a v4 (not a v2 as in TFA) and it's going to have DDR4 ECC, not DDR3 ECC.

I've got a 14 cores / 28 threads Xeon from 2015 that I use as a server at home (ZFS / VMs).

It's really a sweet machine.

For ricing I've got a semi-recent AMD 7700X / DDR5 RAM (from 2023 ?) which is my main machine but the real deal is my old and trusty 10 years old Xeon server.

DDR4 ECC is pricey too atm but a 10 years old Xeon is basically free now.

A 20 cores / 40 threads costs maybe 20 USD (for just the CPU). Slap that in a $100 old HP Z440 workstation and you're good to go for quite a few workloads.

Mine is only on when I'm at my computer: it's not turned on 24/7 but more like 8/7 so the entire "but it consumes energy" point is moot.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#134

Post by ClikeX »

aziis98 wrote: post #39
reidrac wrote: post #27

Earlier quoted context omitted.

> on a decent speed But you said 7-9 tokens/second , that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.

For me, at least for agentic use, you need at least ~40tps. Less might be good only for tasks you could run in the background (like at night maybe). Instead of coding agents like tool for local models I would like to see more "docker ps" like tools where you can queue up tasks that get processes incrementally (maybe when the pc i idle) and are specialized for doing retries and caching as much work as possible to work…

The slower models seem fine for home lab usecases such as processing document transcriptions and tagging them, for example. I don’t need that to be live, it can just churn overnight.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#135

Post by allknowingfrog »

pmarreck wrote: post #121
allknowingfrog wrote: post #60

Earlier quoted context omitted.

We clearly have different goals. I want an LLM to review my code, not the other way around.

I'm sure this exact topic has been argued hundreds of times already on HN, but I think I have a new "possibly agreeable to both sides" perspective on this after having lost man-years to retired corporate code aka "FAIAP, throwaway code" Let LLMs write the corpo code, as it will be unlikely to still be running in 5-10 years. Frontier AI is already at the point where it writes fewer bugs per LOC than humans. By a lot.…

There's a whole spectrum of employment between faceless corporations and personal side projects. AI will replace humans because giant business believe they can do the same work, not because they will actually be able to.

The correctness of an application is limited by your ability to understand and describe what you need. We have a word for an application specification tool so detailed it eliminates all ambiguity. It's called a "programming language".

The mistakes are always in the transfer from human to machine. I still find a high-level programming language to be the best way to express my intent. Humans will make mistakes in the hand-off to AI just like they make mistakes in the hand-off to code, but at least code is deterministic.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#136

Post by hx8 »

RussianCow wrote: post #70
LeifCarrotson wrote: post #38

Earlier quoted context omitted.

I think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare! https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of lik…

Once you've used a model that runs at hundreds of TPS, it's hard to go back. Everything completes so quickly that you can iterate without breaking out of flow state. My biggest gripe with slow (<50tps) LLMs is that I've lost all the mental context I built up by the time it's done, which makes it extremely difficult to explore or iterate on solutions.

I'd rather have slower and better output than worse and faster output.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#137

Post by kevincox »

aziis98 wrote: post #39
reidrac wrote: post #27

Earlier quoted context omitted.

> on a decent speed But you said 7-9 tokens/second , that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.

For me, at least for agentic use, you need at least ~40tps. Less might be good only for tasks you could run in the background (like at night maybe). Instead of coding agents like tool for local models I would like to see more "docker ps" like tools where you can queue up tasks that get processes incrementally (maybe when the pc i idle) and are specialized for doing retries and caching as much work as possible to work…

I think 10 t/s output is usable for many coding workflows if the input speed is much much higher (~100 t/s is a rough minimum line). The low output speed can really hurt for heavy reasoning output but it can still be used to get some jobs done as long as you don't expect highly interactive use.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#138

Post by acchow »

smeeth wrote: post #51
dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

Probably won't have to wait that long. Prism released Bonsai 27B ( https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit ) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one. It is debatable if we will actually need that…

We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#139

Post by kennywinker »

iwontberude wrote: post #110
kennywinker wrote: post #104

Earlier quoted context omitted.

Unless there are major improvements to how much hardware it takes to run a 1T model, this is deeply unrealistic. First because why release hardware that puts your biggest customers (data centers) out of business. Second because as I understand it the data centers have bought up all the high end chip production capacity for at least the next year and unless the bubble pops that'll continue for a while.

Because for the company that will actually do it, their biggest customers aren’t data centers they are iPhone owners.

First off the math doesn’t math. Datacenters are willing to pay $50k for a single high end GPU. If you have unlimited capacity, yeah sell millions for $100 a pop or $10 a pop or whatever the bom cost of a phone GPU would be - but if you have limited capacity, you’re gonna sell all of that to the customer who is willing to pay the most PER UNIT.

Second off, this doesn’t work from a power consumption standpoint. When I run qwen3.6-35b, a far smaller model than op is suggesting, power usage spikes to 150-200W during inference. To fit a 1T model in the palm of my hand, the amount of processing required doesn’t fit the amount of power available.

Now I’m not saying this will never happen - there are some great leads, e.g. burning models directly on to a chip - but op’s scenario is definitely not happening in two years. Maybe 5, a lot more likely 10, unless of course local ai is made illegal

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#140

Post by dwa3592 »

Aurornis wrote: post #129
dwa3592 wrote: post #109

Earlier quoted context omitted.

>>Yes, it’s technically running, but not in a way that would be useful by normal LLM standards. What are the LLM standards? Do you know how many people use perplexity? I know many people who are not software engineers or tech workers and have a LLM subscription for rewriting their stuff (non-native english speakers) in english. There are many use cases for running good models locally. Maybe not for you, but someone m…

I have a free perplexity account from some promotion. Not sure what comparison you’re trying to make because Perplexity’s whole thing is that it’s really fast. It launches the search with parallel agents and then even seems to render some of the output paragraphs with parallel sessions to get the results. Doing the same thing at 7-9 tokens per second, concurrency of 1, would take ages for all of the tool calling and…

Its really easy to argue against local models because when it comes to quality, you can argue using the tokens/sec. and when it comes to speed, you can argue using the parameter count. This is not compared to the frontier stuff but it is the frontier of last year that now runs on a local machine. It was impossible to do this last year.
Post reply on HN