Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

neomindlabs.com

Post reply on HN
Print view View original

211–220 of 227 posts

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#211

Post by pjmlp »

Aurornis wrote: post #86
twsted wrote: post #83

Earlier quoted context omitted.

I think, may be actually wrong, that most of us do not consider running a model locally a way to save money. It is a way not to spread personal info around.

Anyone running LLMs at home will come to that realization quickly, if they’re looking at their power bills. Even feeling the heat output of a computer running at 100% in your office makes it clear. I was responding to a lot of the comments saying this was a reasonable way to avoid paying for tokens or subscriptions. I don’t want anyone getting the wrong idea that this is a way to save money if that’s their priority.

I never get this kind of discussion on HN, many of us live in regions where this is a non issue.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#213

Post by dwa3592 »

mseri wrote: post #196
mseri wrote: post #183

Earlier quoted context omitted.

Nice! I had done the exact same with gemma4 26b, both for my Intel laptop and for my M1 with 8Gb RAM (with also q4 and turboquant). I don’t use it much since there are dumber but way faster models to run, but I should clean up the code and make it available

Ok, done: https://github.com/mseri/zunzuncito My main focus is systems with very low ram. On my M1 with 8Gb or RAM llama.cpp cannot run gemma4 26b, but this implementation works fine with 5-6 tok/s. I also replaced my crap implementation of the oai server following yours, I hope it's all right. I did add a mention to your project and this fact in the README. It is a lot more barebones than what you have, but I have t…

glad to know. thanks :)

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#214

Post by iwontberude »

kennywinker wrote: post #170
andrekandre wrote: post #160

Earlier quoted context omitted.

> Datacenters are willing to pay $50k for a single high end GPU. its true for now, because capital is flowing like a torrent, but how long will that last if returns start to be expected (aka the bubble pops)?

Even if the bubble pops and anthropic and openai et al implode - genie doesn’t go back in the bottle. The usefulness of LLMs for coding is proven, and a chip in a datacenter running 24/7 is always going to be more valuable than in a personal device running occasionally. That doesn’t change until production capacity exceeds the datacenter demand. When that happens, they’ll start selling them down the market until it e…

LLMs for coding is too small of a benefit to justify this investment, the bubble is indeed going to burst. Genie is already on its way back into the bottle.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#215

Post by iwontberude »

kennywinker wrote: post #188
typon wrote: post #177

Earlier quoted context omitted.

There is ton of room for improvement "down there". * Software inference optimizations * Heavy quantization * Chips with hardcoded transformer architecture * Much cheaper HBM * Much sparser models - 1T total with ~1-10B active params e.g. * Not to mention - 2 years of today's frontier models writing RTL and kernels at superhuman levels.

> * Software inference optimizations Absolutely. I'd be surprised if they couldn't 2x performance in the next year. Still doesn't make a 1T model fit on your phone. > * Heavy quantization I think this is a dead end if you're trying to fit a 1T model into a phone. Makes much more sense to train a model that's designed to be small, than train a model that's smart and then quantize it into stupidity. > * Chips with hard…

We dont need god models (1T+) to do dog work. People use far more powerful models than they ever need to, but they don’t even know it yet. They create fake demand with FOMO.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#216

Post by iwontberude »

kennywinker wrote: post #139
iwontberude wrote: post #110

Earlier quoted context omitted.

Because for the company that will actually do it, their biggest customers aren’t data centers they are iPhone owners.

First off the math doesn’t math. Datacenters are willing to pay $50k for a single high end GPU. If you have unlimited capacity, yeah sell millions for $100 a pop or $10 a pop or whatever the bom cost of a phone GPU would be - but if you have limited capacity, you’re gonna sell all of that to the customer who is willing to pay the most PER UNIT. Second off, this doesn’t work from a power consumption standpoint. When I…

You are assuming people need the models they use today. The reality is much much smaller models will suffice (i.e. dont use god models for dog work)

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#217

Post by rvba »

NortySpock wrote: post #115
rvba wrote: post #76

Earlier quoted context omitted.

Thank you for your reply. I use LM studio (local server), but can switch to a different tool. Do you know how to switch it in LM studio? What I see is that: android studio gives "Error: stream failed" and in LM studio server I see it is still working, then says that client (=android studio) disconnected. So I assumed it was a setting in android studio.

Dunno, I have not used either of those. (Had been using zed and ollama, and ollama had plenty of odd defaults that needed fixing) Glancing through the docs, I would be digging down in the config of both Android studio and lm studio for either a TTL or jit auto evict setting, and if you find it, set it to some large number measured in hours? https://developer.android.com/studio/gemini/use-a-local-mode... https://lmstu…

I saw those and sadly none of those works not provides solution to "Error: stream failed" that happens like clockwork.

I found this one:

https://www.jetbrains.com/help/toolbox-app/remote-agent-time...

But it requires me to reinstall android studio because jetbrains toolbox cannot find it... because I installed ir on D:

On a side note Android studio atill creates hundred of megabutea of hidden stuff on C:

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#218

Post by kennywinker »

iwontberude wrote: post #216
kennywinker wrote: post #139

Earlier quoted context omitted.

First off the math doesn’t math. Datacenters are willing to pay $50k for a single high end GPU. If you have unlimited capacity, yeah sell millions for $100 a pop or $10 a pop or whatever the bom cost of a phone GPU would be - but if you have limited capacity, you’re gonna sell all of that to the customer who is willing to pay the most PER UNIT. Second off, this doesn’t work from a power consumption standpoint. When I…

You are assuming people need the models they use today. The reality is much much smaller models will suffice (i.e. dont use god models for dog work)

I’m not assuming that at all, I’m responding to someone suggesting we’ll be able to run 1T models on phones in 2-ish years.

I absolutely agree that models are going to advance on to “edge” hardware over the next few years by becoming small + specialized.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#219

Post by kennywinker »

iwontberude wrote: post #214
kennywinker wrote: post #170

Earlier quoted context omitted.

Even if the bubble pops and anthropic and openai et al implode - genie doesn’t go back in the bottle. The usefulness of LLMs for coding is proven, and a chip in a datacenter running 24/7 is always going to be more valuable than in a personal device running occasionally. That doesn’t change until production capacity exceeds the datacenter demand. When that happens, they’ll start selling them down the market until it e…

LLMs for coding is too small of a benefit to justify this investment, the bubble is indeed going to burst. Genie is already on its way back into the bottle.

I agree it’s too small a benefit to justify the investment, and I agree the bubble will pop. I just don’t think that means hardware prices become sane again for quite a while. I think if you half the price of a server GPU because demand from the big ai companies drops out, we’ll still have a shortage - it’ll just being going into commodity data centers to run open weight models.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#220

Post by iwontberude »

kennywinker wrote: post #218
iwontberude wrote: post #216

Earlier quoted context omitted.

You are assuming people need the models they use today. The reality is much much smaller models will suffice (i.e. dont use god models for dog work)

I’m not assuming that at all, I’m responding to someone suggesting we’ll be able to run 1T models on phones in 2-ish years. I absolutely agree that models are going to advance on to “edge” hardware over the next few years by becoming small + specialized.

Sorry to put words in your mouth, totally agree
Post reply on HN