Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A 10 year old Xeon is all you need

point.free

Post reply on HN
Print view View original

181–190 of 301 posts

Re: A 10 year old Xeon is all you need

#181

Post by cmiles8 »

brookst wrote: post #136
PLenz wrote: post #104

Earlier quoted context omitted.

This. OpenAI and Anthropic are ultimately compute infrastructure plays and not really AI. Everyone will have models, they'll have the ability to run them. This is why the GPU shortage is in their favor.

How does that view align with Anthropic leasing data centers from others? I don’t know OpenAI’s infra, but to the extent they are buying GPUs and building data centers with their own money, that sounds like a bad move. Satya has mismanaged the AI transition in many ways, but one thing he got right is that models are commodities, and the value is in applications that apply them to create user benefit. I agree that any…

Then they go bankrupt.

Re: A 10 year old Xeon is all you need

#182

Post by quietsegfault »

dangus wrote: post #129
RetroTechie wrote: post #106

Earlier quoted context omitted.

How many kWh to fabricate a brand new machine better suited to the task? As long as performance is useable (apply your own metrics!), pulling it from existing hardware is likely the option with the lower eco footprint. Also: chances are it'll only be used for this purpose occasionally, and/or for a short while. In that scenario [fabricating new hardware] always has the bigger eco footprint.

I don’t know why you’d assume that an older system is lower footprint. If you’ve got something consuming 100 watts average over your 24 hour period, and your electricity costs 20 cents per kWh, you’re already spending almost as much as a Claude subscription. Just on electricity, this assumes your hardware never fails and you never incur any additional costs. There’s a big reason why newer more efficient hardware is i…

I have two LARGE Xeon systems of this era that I used to use when I was heavily involved with Kubernetes and needed to build out a home lab. One is 2x Xeon w/ 256 GB of ram, and one is 1x Xeon w/ 512GB of ram. Both are slow as dogs, and both of them take up at least 150+ watts with only one power supply. My 12th gen Intel Nuc is so, so much faster and efficient. I'm recycling the Xeon systems.

Re: A 10 year old Xeon is all you need

#183

Post by greenavocado »

montroser wrote: post #97

Result is ~12 tokens per second, as reported by OP down in these comments here. An impressive effort, and better than I would have thought possible on this hardware -- but still pretty far short of what one needs for an satisfactory interactive session.

I was about to ask that

Re: A 10 year old Xeon is all you need

#184

Post by vetrom »

arpinum wrote: post #25
cafkafk wrote: post #2

Hi HN. I wrote this post after getting frustrated by the lack of ways to run the new Gemma 4 Drafter models, and mainstream tools not prioritizing this, and hiding all the performance levers. I ended up getting a modern 26B MoE model (Gemma 4) running at reading speed on an old recycled server with a single Xeon E5-2620 v4 and 128GB of DDR3 RAM (and no GPU). It took a lot of work, but it actually worked out somehow.…

How many watts is that setup? Cool you got it to work, but maybe only useful for vintage / retro computing rather than practical if the energy consumption makes it economically wasteful.

IDK about OPs setup, but I run a pile of E5-2683v4 Xeon recycled servers for Ceph and self hosted business SaaS usage.

One node's ipmitool sensor report (and self-monitoring PSU, so grain of salt, but my UPS side monitoring tracks closely), reports 250-300w average power use. This though, mind you is for running 22 spinning disks, 2 SAS/SATA SSDs, and 4 NVME ssds, and 768GB of DDR4.

Mid-gen 2015ish Xeons were not great at power reduction, but if you are pegging the cores, they were never particularly slow, and they did have lots of PCIe lanes. This boils down to the CPU/mobo itself not being that big a cost floor, especially if you have high utilization rates.

As a comparison, my main desktop development machine, running a Threadripper 9970X, 128GB of DDR5, a RDNA4 GPU, and a small pile of NVME drives has a power floor of roughly 250W. Some CPU centric workloads you'll definitely lose out on on the older gens of machines, but they are by no means impractical.

Maybe for a desktop usecase they are absolutely suboptimal nowadays, but for a lot of realworld usecases I would say they're still relevant.

---

Like the author posts for the LLM usecase, I think optimizing the hardware choice to the application and not leaving levers unpulled is a big key, especially considering how wide a variety of bandwidth/power draw/peak frequency/corecount SKUs exist in the Xeon lines. Without knowing what you intend to run and fitting the correct processor to it, you will end up with a disappointingly poor environment fit.

Re: A 10 year old Xeon is all you need

#185

Post by DrScientist »

twoodfin wrote: post #72
MAXPOOL wrote: post #46

Earlier quoted context omitted.

Things you are not supposed to talk about: - There is no "moat" (lasting, easy-to-defend technological edge) in AI model businesses. There are just short-term advantages. - An AI business is a capital-intensive business, just like old factories. Data centers are expensive, models are energy-hungry, and the hardware inside must be replaced every 3–4 years. - Smaller, specialized models eat margins from below. Transcri…

All true. It does seem like the structural characteristics we’ve observed so far suggest there is a kind of flywheel from short-term to long-term advantage due to the capital requirements at various levels. If you’re Nvidia, making the best GPUs today, the expanding wavefront of demand is consuming them with volume and margins to give you a huge edge in building out the best next generation of GPUs. Similar to how th…

I can see the fly wheel in action for Nvidia[1], but in terms of model building - I think the companies that have the advantage here are not Anthropic or OpenAI, but rather companies with substantial revenues from other sources - Google is the obvious player here - reported to be planning on spending 185 billion this year without having a raise a dime from the markets, but there are plenty of other companies - like Meta or Alibaba who can easily fund the longer game from existing revenues.

Re: A 10 year old Xeon is all you need

#186

Post by ryandrake »

I've got an old HP Z-620 workstation with dual E5-2697 v2 CPUs (24 cores total, 48 threads @ 2.7GHz) and 128GB of DDR3 RAM. The docs say it supports up to 192GB, but I wasn't able to get it to POST with all the RAM slots full.

It's still a "homelab" beast and does great with development and GIS/Mapping applications. I was not able to figure out how to run AI workloads on it with decent performance, however, so I finally broke down and got a dedicated GPU for it. It's pretty great what can still be done with older hardware.

Re: A 10 year old Xeon is all you need

#187

Post by throwaway2037 »

jansommer wrote: post #73
throwaway2037 wrote: post #65

Earlier quoted context omitted.

> The E5-2620 v4 is great. Have been using it for 10 years now. 10 years? Damn, that is a long time. I always assumed that heat-induced damage will kill a CPU after a certain amount of time (5-7 years). Am I wrong here? I assume yes. Or are CPUs must stronger/tougher than the bad old days?

A quick search on Xeon production yields that it goes through a rather rigorous testing. I wouldn't be surprised that server cpu's in a desktop pc works longer. I can't overclock it either, and that probably helps with its lifespan as well. But yeah, the fact that it actually powers on when i click the button and isn't a limiting factor after 10 years is quite something.

You raise two very good points that I didn't think about: (1) better binning/testing, (2) no overclocking. Keep rockin' that elderly Xeon!

Re: A 10 year old Xeon is all you need

#188

Post by FpUser »

ryandrake wrote: post #186

I've got an old HP Z-620 workstation with dual E5-2697 v2 CPUs (24 cores total, 48 threads @ 2.7GHz) and 128GB of DDR3 RAM. The docs say it supports up to 192GB, but I wasn't able to get it to POST with all the RAM slots full. It's still a "homelab" beast and does great with development and GIS/Mapping applications. I was not able to figure out how to run AI workloads on it with decent performance, however, so I fina…

I self host on old HP Z-840 with 2x3.6 GHz Xeons 24 total cores, and 512 GB RAM. Cost me peanuts used and works like a charm for many years already

Re: A 10 year old Xeon is all you need

#190

Post by fooker »

wazoox wrote: post #92
fooker wrote: post #44

Earlier quoted context omitted.

What are the tasks that do well with 8-10 t/s ?

The sort of task you don't expect to end immediately. If extracting data from a bunch of PDFs takes 1 hour or the whole night, that doesn't make much difference to me. It's not fast enough for auto completion and slightly too slow for chat (but bearable IMO).

Running a local llm at 10 t/s overnight to extract data from a few PDFs will burn more in electricity than paying cents for the hosted kimi models.

You can (sometimes) break even if you have a workstation GPU.

Post reply on HN