Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A 10 year old Xeon is all you need

point.free

Post reply on HN
Print view View original

251–260 of 301 posts

Re: A 10 year old Xeon is all you need

#251

Post by hualapais »

Went this route after hemming and hawing over a Mac Studio Pro for some time. Eventually bought and configured a headless HP Z620 with 192 GB of ECC RAM and dual Xeon E5-2680 v2 processors, an Optane AIC, two P102-100s with 10 GB VRAM each, and a minimal bootable SDD running Debian 12.6 with an older, locked version of CUDA that supports the Pascal cards. Run it remotely from the basement via AMT/meshcommander. Just fire up llama.cpp and its front end and connect over the local network. Currently playing with Talkie, Qwen 3.6 27b, and medgemma, but have had good luck with GGUF performance in general after selecting an appropriate quant. Total cost was under $500, but I bought the server via eBay last year; things may be different now.

Details aside, the hope is that ternary LLMs blossom in the coming months and this old hardware can eventually host some very dense models full of factual information, perhaps even larger than the GPU RAM and spilling over to the Optane for IO. Speed would be less important than general factual knowledge. The plan would be to configure then mothball the machine in a Faraday trashcan in the basement, retaining it as a possible "rebuild civilization" oracle should the world fall apart. Of course, power would be an issue in such a scenario, but for how cheap this hardware is and how often AI seems to be practically useful in its latest iterations, why not...

Re: A 10 year old Xeon is all you need

#252

Post by dangus »

gnerd00 wrote: post #208
quietsegfault wrote: post #182

Earlier quoted context omitted.

I have two LARGE Xeon systems of this era that I used to use when I was heavily involved with Kubernetes and needed to build out a home lab. One is 2x Xeon w/ 256 GB of ram, and one is 1x Xeon w/ 512GB of ram. Both are slow as dogs, and both of them take up at least 150+ watts with only one power supply. My 12th gen Intel Nuc is so, so much faster and efficient. I'm recycling the Xeon systems.

Xeon is a group of products with really varying specs. There is no indication of which XEONs. Also new consumer CPUs often have really small internal caches.

The Xeon processor in use by the OP of this article claims to have 20MB of Intel “Smart Cache.”

An Apple M4 chip in a Mac mini has 16MB on the P-cores and 4MB on the E-cores.

Depending on use case, AMD 3D V-cache at almost 100MB could also work out quite well.

So really, if you wait long enough, consumer chips end up with a pretty similar amount of cache.

Re: A 10 year old Xeon is all you need

#254

Post by dangus »

ThatMedicIsASpy wrote: post #218
dangus wrote: post #129

Earlier quoted context omitted.

I don’t know why you’d assume that an older system is lower footprint. If you’ve got something consuming 100 watts average over your 24 hour period, and your electricity costs 20 cents per kWh, you’re already spending almost as much as a Claude subscription. Just on electricity, this assumes your hardware never fails and you never incur any additional costs. There’s a big reason why newer more efficient hardware is i…

The reason more performance/watt is in demand because a datacenter can't suddenly draw twice as much power.

Or because I don’t want my homelab to spike my electricity bill and give me a loud hot closet.

Re: A 10 year old Xeon is all you need

#255

Post by coldcity_again »

rythie wrote: post #114
coldcity_again wrote: post #76

This is great work. I'd love if anyone knows how I might fare with an old Dell R710 with 2 x Xeon 5600 (12 cores total) and 96Gb of DDR3.

I don’t think it would work as well as there is no AVX or AVX2 on those older CPUs unfortunately.

Thanks very much. I'd forgotten that these were Westmere generation! Experimenting anyway; at least the RAID controller is behaving, and Ubuntu 26.04 LTS has gone on cleanly.

Re: A 10 year old Xeon is all you need

#256

Post by throwawayffffas »

boutell wrote: post #59
ekianjo wrote: post #37

Earlier quoted context omitted.

20 tokens per second for eval time is the killer here. It means you can't use this to process any meaningful amount of text. A GPU typically processes close to 1000 tokens/s during eval.

I'm pretty sure eval time is token generation time where it's actually outputting new tokens. If you're getting a thousand per second on that, I'd love to know on what.

He meant prompt eval time, but have a look at these guys: https://www.youtube.com/watch?v=ndSA9T5yvmM

Over 2500 tokens per second on a single request. With 8 MI300X.

Re: A 10 year old Xeon is all you need

#257

Post by philipkglass »

materielle wrote: post #250
cmiles8 wrote: post #84

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.

One thing I don’t quite understand: Wouldn’t it be in Amazon’s interest to run open models and sell time slots at around the cost of running them? My only guess for why they don’t is that AI labs are currently selling their models at a huge loss, so this isn’t worth Amazon spending low-margin compute on compared to other higher margin products. What I’m getting at, is maybe we won’t even need to run the models locall…

AWS Bedrock offers a mixture of proprietary and open-weights models (DeepSeek, Nemotron, gpt-oss, etc.):

https://docs.aws.amazon.com/bedrock/latest/userguide/model-c...

Re: A 10 year old Xeon is all you need

#258

Post by _hyn3 »

gdjdhdheb wrote: post #36
cafkafk wrote: post #2

Hi HN. I wrote this post after getting frustrated by the lack of ways to run the new Gemma 4 Drafter models, and mainstream tools not prioritizing this, and hiding all the performance levers. I ended up getting a modern 26B MoE model (Gemma 4) running at reading speed on an old recycled server with a single Xeon E5-2620 v4 and 128GB of DDR3 RAM (and no GPU). It took a lot of work, but it actually worked out somehow.…

You sure you got DDR3 .. I have 2 e5 v4 rigs at home and both have ddr4 ... Unless I am wrong and 2011-3 supports ddr3 and ddr4

You're right - the article says 'CPU: Intel Xeon E5-2620 v4 @ 2.10 GHz' but also says DDR3. And the specs page for that CPU (https://www.intel.com/content/www/us/en/products/sku/92986/i...) clearly says the 2620 v4 is DDR4.

E5 CPUs have their supported RAM right on the Intel ARK pages, but short version:

E5-xxxxx v1 and v2 are all DDR3

E5-xxxxx v3 and v4 are all DDR4

Not sure why Intel didn't just cut new model numbers instead of keeping them all as "e5"

More concrete example for E5-2660 (great processor) showing v1 and v2 support DDR3, while v3 and v4, DDR4 (again, different motherboards)

DDR3 v1: https://www.intel.com/content/www/us/en/products/sku/64584/i...

DDR3 v2: https://www.intel.com/content/www/us/en/products/sku/75272/i...

DDR4 v3: https://www.intel.com/content/www/us/en/products/sku/81706/i...

DDR4 v4: https://www.intel.com/content/www/us/en/products/sku/91772/i...

This also means that you need to know the processor your motherboard supports (or, easier, probably RAM) before putting in an order to upgrade the processor. (These processors are incredibly cheap, less than $10 for something that might have cost literally thousands ten years ago, so worthwhile to spend a few minutes and pick out your favorite based on cores, watts, Ghz, etc.)

(Another commenter says that there are some motherboards that accept v3/v4 but also can run slower DDR3 RAM. That's new to me and quite cool - DDR3 is extremely cheap, even now. I did find these motherboards on aliexpress, too: https://www.aliexpress.us/w/wholesale-XD3-motherboard.html?s... and one clearly says v3/v4 cpu's with DDR3 RAM. That could be very useful although memory speeds are slower since CPU performance can be boosted with v3/v4.)

v1: https://www.intel.com/content/www/us/en/ark/products/series/...

v2: https://www.intel.com/content/www/us/en/ark/products/series/...

v3: https://www.intel.com/content/www/us/en/ark/products/series/...

v4: https://www.intel.com/content/www/us/en/ark/products/series/...

Re: A 10 year old Xeon is all you need

#259

Post by billfor »

cmiles8 wrote: post #84

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.

It might just shift who is buying memory, from large corporations to billions of individuals.
Post reply on HN