Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

neomindlabs.com

Post reply on HN
Print view View original

61–70 of 227 posts

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#61

Post by sosodev »

hagen8 wrote: post #44

Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally. If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider. I estimate that the server consumes probably around 500W during inference. In Germany where 1kwh cost arou…

Of course efficiency matters, but a lot of people either have cheap electricity or efficient hardware. My AMD strix halo home server can serve Gemma4-26B at like 70 TPS (rough estimate, I don’t remember the exact speed buts its fast af) while only using 100W.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#62

Post by albrewer »

hagen8 wrote: post #44

Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally. If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider. I estimate that the server consumes probably around 500W during inference. In Germany where 1kwh cost arou…

Not OP, but your math is a bit off - I have solar panels :)

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#63

Post by bischofs »

hagen8 wrote: post #44

Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally. If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider. I estimate that the server consumes probably around 500W during inference. In Germany where 1kwh cost arou…

I don't pay anywhere near 0.30usd in the US - I pay half that off peak and can buy 1000$ worth of batteries to load up on super off peak (0.11usd). Also the inference providers are fighting over market share with huge debt loads so they are definitely going to go up in price.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#64

Post by superkuh »

fouc wrote: post #33
neomindryan wrote: post #17

Earlier quoted context omitted.

hey, I’m the author. That box has 384gb, but loading the model “only” uses about 80gb.

any reason you went with q8 over q4? I'm wondering if q4 would run noticeably faster or not.

Such a system is RAM bandwidth limited and not compute limited Switching to q4 from q8 would decrease the amount of data needing to be loaded by half. The token generation rate would nearly double. But generally if you can do q6 or q8 and you have enough RAM you really should. Even if it's slower.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#65

Post by NortySpock »

rvba wrote: post #10

Apologies for asking here but literally nobody knows: Android studio connected to a local model disconnects automatically after 10 minutes. How set this limit to 12 hours or remove it completely? I could run my LM studio model all night... but I cant, since Android studio times out after a hard limit of 10M. This is not related to number of tokens. I tried Googling, searching for settings in Android studio, even crea…

If the local model is served via ollama, there's a default timeout of 10 minutes , which can be adjusted either per-call , or (as I did) in the systemd service environment variables

https://docs.ollama.com/faq#how-do-i-keep-a-model-loaded-in-...

You didn't specify what was serving your local model.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#66

Post by kylemaxwell »

hagen8 wrote: post #44

Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally. If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider. I estimate that the server consumes probably around 500W during inference. In Germany where 1kwh cost arou…

You pay 3x as much for electricity as I do, so the math here is going to work out very differently depending on a lot of factors.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#67

Post by amanaplanacanal »

bischofs wrote: post #63
hagen8 wrote: post #44

Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally. If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider. I estimate that the server consumes probably around 500W during inference. In Germany where 1kwh cost arou…

I don't pay anywhere near 0.30usd in the US - I pay half that off peak and can buy 1000$ worth of batteries to load up on super off peak (0.11usd). Also the inference providers are fighting over market share with huge debt loads so they are definitely going to go up in price.

Yeah I had to check, I'm paying 0.08usd per kwh. This is in the US PNW with quite a bit of local hydro power.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#68

Post by nolok »

allknowingfrog wrote: post #60
LeifCarrotson wrote: post #38

Earlier quoted context omitted.

I think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare! https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of lik…

We clearly have different goals. I want an LLM to review my code, not the other way around.

It's still the same thing, you can ask it to do a full on report give explanation and details be thorough and then go do something else, another task a lunch break whatever and it will be done when you're back

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#69

Post by brewtide »

amanaplanacanal wrote: post #67
bischofs wrote: post #63

Earlier quoted context omitted.

I don't pay anywhere near 0.30usd in the US - I pay half that off peak and can buy 1000$ worth of batteries to load up on super off peak (0.11usd). Also the inference providers are fighting over market share with huge debt loads so they are definitely going to go up in price.

Yeah I had to check, I'm paying 0.08usd per kwh. This is in the US PNW with quite a bit of local hydro power.

It's all relative. On the opposite coast, Maine it is ~ $0.28 cents kwh including getting it there. (~ 50% energy, 50% delivery). It's too darn expensive here.

Re: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

#70

Post by RussianCow »

LeifCarrotson wrote: post #38
reidrac wrote: post #27

Earlier quoted context omitted.

> on a decent speed But you said 7-9 tokens/second , that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.

I think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare! https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of lik…

Once you've used a model that runs at hundreds of TPS, it's hard to go back. Everything completes so quickly that you can iterate without breaking out of flow state. My biggest gripe with slow (<50tps) LLMs is that I've lost all the mental context I built up by the time it's done, which makes it extremely difficult to explore or iterate on solutions.
Post reply on HN