A 10 year old Xeon is all you need • Hacker News Board

Re: A 10 year old Xeon is all you need

#51

Post by Sweepi » Mon, Jun 01, 2026, 10:23 AM UTC

cafkafk wrote: post #2

Hi HN. I wrote this post after getting frustrated by the lack of ways to run the new Gemma 4 Drafter models, and mainstream tools not prioritizing this, and hiding all the performance levers. I ended up getting a modern 26B MoE model (Gemma 4) running at reading speed on an old recycled server with a single Xeon E5-2620 v4 and 128GB of DDR3 RAM (and no GPU). It took a lot of work, but it actually worked out somehow.…

"-t 8 matches physical cores. The machine has 16 SMT threads but only 8 cores. On a memory-bound workload, oversubscribing threads adds scheduling cost without adding throughput: the cores are waiting on DDR3, not on each other."

But ... isnt that a classic use case for SMT? Giving T1 sth. to do while T0 is waiting on DDR(3) and vise-versa?

I also dont understand the explanation of "--cpu-moe". If an expert has ~ 4.0 GiB of Parameters, why does optimizing the sequence of experts minimize cash trashing? With 20 MiB of L3 Cash vs 4.0 GiB of Parameters, it wont cash any noticeable amount of the Parameters, will it?

As mentioned by others, only some Intel Xeon E5-2xxx v4 did support DDR3, and according to Intel, the E5-2620 v4 is not one of them.

Re: A 10 year old Xeon is all you need

#52

Post by jansommer » Mon, Jun 01, 2026, 10:27 AM UTC

The E5-2620 v4 is great. Have been using it for 10 years now. Wanted to upgrade until I saw current prices. I have 64 GB ddr4. Paired it with rx 9060 xt 16 GB and games run as fast as ever. Perhaps the cpu is a slight bottleneck in DOOM The Dark Ages, but i'm at 60 fps, so no problem. Light llm on the gpu is a nobrainer, and it's cool to see that things can be tuned to run ok on the cpu. I bought 2667 v4 a month ago for 30$. I'd expect it to give a decent performance boost but I just haven't had the need for it yet, but pushing into llm like in the article I'd probably upgrade because 2667 can handle slightly faster ram.

Re: A 10 year old Xeon is all you need

#53

Post by bflesch » Mon, Jun 01, 2026, 10:30 AM UTC

bflesch wrote: post #22

Might consider going for even older CPUs which don't have the Intel ME ring -3 thing which is full of backdoors

I appreciate the downvotes without any reasoning. It's a fact that newer Intel CPUs have Intel ME which was not in older CPUs and significantly increases attack surface if you are not living in a five eyes state.

Re: A 10 year old Xeon is all you need

#54

Post by deng » Mon, Jun 01, 2026, 10:32 AM UTC

Nice post and technically impressive work. I agree we need to understand the build pipeline and be able to do things locally. However, depending on your electricity cost, it might not make sense financially. These old servers are not energy efficient at all (I'm guessing that old Xeon server will easily pull 200W on load), and that model is currently at 0.1$/0.3$ per 1M tokens (with 76 tps and 262k context) in Openrouter (also, these servers are LOUD).

EDIT: I stand corrected, 200W is apparently way too high of an estimate. I used to run a bunch of old Xeon servers and they slurped watts like crazy, but I can't remember which ones exactly those were.

Re: A 10 year old Xeon is all you need

#55

Post by jansommer » Mon, Jun 01, 2026, 10:36 AM UTC

deng wrote: post #54

Nice post and technically impressive work. I agree we need to understand the build pipeline and be able to do things locally. However, depending on your electricity cost, it might not make sense financially. These old servers are not energy efficient at all (I'm guessing that old Xeon server will easily pull 200W on load), and that model is currently at 0.1$/0.3$ per 1M tokens (with 76 tps and 262k context) in Openro…

It should be closer to 85W on load. And it's incredibly silent on even a low end cooler. I rarely get above 50° Celcius.

Re: A 10 year old Xeon is all you need

#56

Post by rvba » Mon, Jun 01, 2026, 10:39 AM UTC

As someone doing this for fun on a windows 11 machine (96gb ram, 5090 24gb) I wonder if I need any flags to keep the model in memory and avoid swapping to ssd?

I use LM studio and qwen3.5 35B - but never figured out if it is swapping or not.

Om am unrelated note, does anyone know a model that can help with this use case:

https://news.ycombinator.com/item?id=48301635

Re: A 10 year old Xeon is all you need

#57

Post by zamadatix » Mon, Jun 01, 2026, 10:44 AM UTC

Sweepi wrote: post #51

cafkafk wrote: post #2

Hi HN. I wrote this post after getting frustrated by the lack of ways to run the new Gemma 4 Drafter models, and mainstream tools not prioritizing this, and hiding all the performance levers. I ended up getting a modern 26B MoE model (Gemma 4) running at reading speed on an old recycled server with a single Xeon E5-2620 v4 and 128GB of DDR3 RAM (and no GPU). It took a lot of work, but it actually worked out somehow.…

"-t 8 matches physical cores. The machine has 16 SMT threads but only 8 cores. On a memory-bound workload, oversubscribing threads adds scheduling cost without adding throughput: the cores are waiting on DDR3, not on each other." But ... isnt that a classic use case for SMT? Giving T1 sth. to do while T0 is waiting on DDR(3) and vise-versa? I also dont understand the explanation of "--cpu-moe". If an expert has ~ 4.0…

> But ... isnt that a classic use case for SMT? Giving T1 sth. to do while T0 is waiting on DDR(3) and vise-versa?

Waiting in terms of latency. When the bus is mostly empty and it takes a while to make a round trip it's great to try to find a few extra passengers to put on it. When the buses are all completely full adding the extra riders just makes the bus stop that much more chaotic.

Re: A 10 year old Xeon is all you need

#58

Post by mft_ » Mon, Jun 01, 2026, 10:46 AM UTC

anon-3988 wrote: post #23

cafkafk wrote: post #6

Earlier quoted context omitted.

> (purple on black is really hard to read) Noted, and agree (it looks like it has also already been clicked, which I dislike). I honestly I need to redo the themes. > You say it runs "at reading speed". Have you benchmarked it? At some point a few weeks ago, yes I think so, but I didn't write it down for some reason... so I'll have to find a time when it's not busy and do it again without a noisy system. Right now th…

I am pretty sure llamacpp have their own benchmarking binary that you can use.

llama-bench is part of the llama-cpp package, but from recent experimentation, the settings it is able to (or is documented to?) accept lag behind somewhat. Not sure whether it would accept all of the esoteric settings in the article?

Re: A 10 year old Xeon is all you need

#59

Post by boutell » Mon, Jun 01, 2026, 10:47 AM UTC

ekianjo wrote: post #37

cafkafk wrote: post #6

Earlier quoted context omitted.

> (purple on black is really hard to read) Noted, and agree (it looks like it has also already been clicked, which I dislike). I honestly I need to redo the themes. > You say it runs "at reading speed". Have you benchmarked it? At some point a few weeks ago, yes I think so, but I didn't write it down for some reason... so I'll have to find a time when it's not busy and do it again without a noisy system. Right now th…

20 tokens per second for eval time is the killer here. It means you can't use this to process any meaningful amount of text. A GPU typically processes close to 1000 tokens/s during eval.

I'm pretty sure eval time is token generation time where it's actually outputting new tokens. If you're getting a thousand per second on that, I'd love to know on what.

Re: A 10 year old Xeon is all you need

#60

Post by deng » Mon, Jun 01, 2026, 10:48 AM UTC

jansommer wrote: post #55

deng wrote: post #54

Nice post and technically impressive work. I agree we need to understand the build pipeline and be able to do things locally. However, depending on your electricity cost, it might not make sense financially. These old servers are not energy efficient at all (I'm guessing that old Xeon server will easily pull 200W on load), and that model is currently at 0.1$/0.3$ per 1M tokens (with 76 tps and 262k context) in Openro…

It should be closer to 85W on load. And it's incredibly silent on even a low end cooler. I rarely get above 50° Celcius.

OK, then you're in luck. I had a bunch of old 1U rack servers and even in the next room it was too annoying to run them (they had a bunch of 40mm fans which always ran at full speed, because in a server room, no one can hear you scream).