Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

neomindlabs.com

Post reply on HN
Print view View original

121–130 of 227 posts

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#121

Post by pmarreck »

allknowingfrog wrote: post #60
LeifCarrotson wrote: post #38

Earlier quoted context omitted.

I think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare! https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of lik…

We clearly have different goals. I want an LLM to review my code, not the other way around.

I'm sure this exact topic has been argued hundreds of times already on HN, but I think I have a new "possibly agreeable to both sides" perspective on this after having lost man-years to retired corporate code aka "FAIAP, throwaway code"

Let LLMs write the corpo code, as it will be unlikely to still be running in 5-10 years. Frontier AI is already at the point where it writes fewer bugs per LOC than humans. By a lot.

Go ahead and do your bespoke coding on your side-project loves and core libraries... The stuff that will last, anyway.

But if you're working for a corpo and still doing bespoke... That's... not gonna last, I'm afraid. Well, either you remaining there, or that, as it were.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#122

Post by dwa3592 »

felineflock wrote: post #117
dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

That is awesome! I am curious about the decision to not use GPU since this is for Apple Silicon. Wouldn't the GPU potentially accelerate the DeltaNet/attention layers and matrix multiplication in general?

It will, but the process at this point is SSD bound rather than compute bound. On a bigger machine, Apple silicon must help but I don't have a bigger machine. I can think about this more and will make changes if that helps.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#123

Post by embedding-shape »

dwa3592 wrote: post #28
embedding-shape wrote: post #23

Earlier quoted context omitted.

> I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that, seems there was another collaborator ;) Show something you've built with the model+tooling instead, truly dogfood it. I'm sure you'll discover things along the way too!

>>It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that that would have run into a race condition unfortunately ;) but there is a sample landing page + a python function on the repo which shows what the model produced. my goal is to integrate the local model in my workflow so that claude/OAI can call this model for basic stuff.

> that would have run into a race condition unfortunately ;)

Not really, you start small, bootstrap as soon as you can, and off you go. Requires a good model though ;)

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#124

Post by Wowfunhappy »

reidrac wrote: post #27
dwa3592 wrote: post #21

I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

> on a decent speed But you said 7-9 tokens/second , that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.

How fast does a human write code?

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#125

Post by sigmoid10 »

Wowfunhappy wrote: post #124
reidrac wrote: post #27

Earlier quoted context omitted.

> on a decent speed But you said 7-9 tokens/second , that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.

How fast does a human write code?

It's fast if you want to automate things that run independently or overnight. It's slow if you want to iterate code together with it.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#126

Post by fwip »

allknowingfrog wrote: post #88
nolok wrote: post #68

Earlier quoted context omitted.

It's still the same thing, you can ask it to do a full on report give explanation and details be thorough and then go do something else, another task a lunch break whatever and it will be done when you're back

How do you maintain a flow state during a lunch break? I'm looping with Claude on a scale of minutes. While you're waiting, I'm iterating.

You don't "maintain flow." You eat lunch.

I swear, tech culture has gotten people wanting to work for the machines, rather than the other way round.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#127

Post by embedding-shape »

smith7018 wrote: post #31
embedding-shape wrote: post #23

Earlier quoted context omitted.

> I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that, seems there was another collaborator ;) Show something you've built with the model+tooling instead, truly dogfood it. I'm sure you'll discover things along the way too!

Nothing says they're using Qwen for local development. They could be using it to for conversations, knowledge, or "creative writing."

> Nothing says they're using Qwen for local development

I know! That's my point! You're a poor salesman of a coding environment/tool if you don't even use it yourself for coding...

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#128

Post by allknowingfrog »

vitally3643 wrote: post #99
allknowingfrog wrote: post #88

Earlier quoted context omitted.

How do you maintain a flow state during a lunch break? I'm looping with Claude on a scale of minutes. While you're waiting, I'm iterating.

This is like comparing a hammer to a screwdriver and feeling smug because you can hammer nails faster than someone else can drive screws. These are fundamentally different tools for entirely different applications. They only look similar to people who don't understand the tools or their purpose.

This thread started with me saying "we clearly have different goals" and then being told that I just need to hold the screwdriver differently...

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#129

Post by Aurornis »

dwa3592 wrote: post #109
Aurornis wrote: post #75

Earlier quoted context omitted.

> I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. This prediction alone isn’t useful at all without a bound on speed and maybe quantization. You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because the…

>>Yes, it’s technically running, but not in a way that would be useful by normal LLM standards. What are the LLM standards? Do you know how many people use perplexity? I know many people who are not software engineers or tech workers and have a LLM subscription for rewriting their stuff (non-native english speakers) in english. There are many use cases for running good models locally. Maybe not for you, but someone m…

I have a free perplexity account from some promotion. Not sure what comparison you’re trying to make because Perplexity’s whole thing is that it’s really fast. It launches the search with parallel agents and then even seems to render some of the output paragraphs with parallel sessions to get the results.

Doing the same thing at 7-9 tokens per second, concurrency of 1, would take ages for all of the tool calling and subsequent processing.

It wouldn’t compare in any meaningful way, because perplexity delivers instant results. That’s what I meant by modern standards of LLM usefulness.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#130

Post by allknowingfrog »

fwip wrote: post #126
allknowingfrog wrote: post #88

Earlier quoted context omitted.

How do you maintain a flow state during a lunch break? I'm looping with Claude on a scale of minutes. While you're waiting, I'm iterating.

You don't "maintain flow." You eat lunch. I swear, tech culture has gotten people wanting to work for the machines, rather than the other way round.

This was a discussion about LLM usage patterns. I'm not opposed to lunch breaks. I'm opposed to being required to take the equivalent of 12 lunch breaks a day while I wait for slow responses.
Post reply on HN