so how many tokens/s do you get, pp and tg? did I miss it in the article?
A 10 year old Xeon is all you need
241–250 of 301 posts
Re: A 10 year old Xeon is all you need
#242Earlier quoted context omitted.
It should be closer to 85W on load. And it's incredibly silent on even a low end cooler. I rarely get above 50° Celcius.
OK, then you're in luck. I had a bunch of old 1U rack servers and even in the next room it was too annoying to run them (they had a bunch of 40mm fans which always ran at full speed, because in a server room, no one can hear you scream).
Small fans need to spin faster so these can be very high pitch even if you stuff some Noctua 40mm fans into it.
Re: A 10 year old Xeon is all you need
#243Earlier quoted context omitted.
I don’t know why you’d assume that an older system is lower footprint. If you’ve got something consuming 100 watts average over your 24 hour period, and your electricity costs 20 cents per kWh, you’re already spending almost as much as a Claude subscription. Just on electricity, this assumes your hardware never fails and you never incur any additional costs. There’s a big reason why newer more efficient hardware is i…
You mention lower footprint but then make a cost comparison against Claude subscription pricing. Claude subscription pricing is a broken way to consider footprint.
You can call it whatever you want, money is money, and money spent on energy is footprint.
Re: A 10 year old Xeon is all you need
#244We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.
More likely we will have a compute device like NAS or something which will run one good model locally for all the house members just like we have one wifi router in every house. Nvidia can invest in building such a device as well as the models and make money on the hardware.
Re: A 10 year old Xeon is all you need
#245Earlier quoted context omitted.
I have two LARGE Xeon systems of this era that I used to use when I was heavily involved with Kubernetes and needed to build out a home lab. One is 2x Xeon w/ 256 GB of ram, and one is 1x Xeon w/ 512GB of ram. Both are slow as dogs, and both of them take up at least 150+ watts with only one power supply. My 12th gen Intel Nuc is so, so much faster and efficient. I'm recycling the Xeon systems.
Xeon is a group of products with really varying specs. There is no indication of which XEONs. Also new consumer CPUs often have really small internal caches.
E5-2690s in my case.
Re: A 10 year old Xeon is all you need
#246The E5-2620 v4 is great. Have been using it for 10 years now. Wanted to upgrade until I saw current prices. I have 64 GB ddr4. Paired it with rx 9060 xt 16 GB and games run as fast as ever. Perhaps the cpu is a slight bottleneck in DOOM The Dark Ages, but i'm at 60 fps, so no problem. Light llm on the gpu is a nobrainer, and it's cool to see that things can be tuned to run ok on the cpu. I bought 2667 v4 a month ago…
I'm on a dual-E5 2667-v4 / 256 GB DDR4 Z640 with a 1080ti that I picked up all the various pieces for (aside from SSDs) for less than $500 total in the first half of 2025 (case, PSU, riser board included). I'm still kind of blown away by what you can find aftermarket / secondhand!
I also had no idea RAM and GPU costs would explode they way they did, just happened to do it the right time. I might try to grab a ~$300 3080 on Ebay and sell the 1080ti, but otherwise it's been a great upgrade -- it sucks electricity like Coca Cola, but otherwise performs fantastic as a workstation, and I'm just gonna drive it til the wheels fall off.
Re: A 10 year old Xeon is all you need
#247I bought one AMD MI50 32GB back then when they were sold rather cheap (around $150-$170). it can easily generate over 70 tokens per second for gemma 4 26B moe model (q4).
I have no doubt that we will have another wave of cheap retired server gpus just like before. And that is the time when everyone will have their own models at their home.
Or we can just buy the newest medusa halo mini pc. they will be pretty decent, too, albeit pricey.
Re: A 10 year old Xeon is all you need
#248Earlier quoted context omitted.
Phones are also a terrible place to run a radio, but there's a huge amount of benefit in figuring out how to do so.
That level of local AI is also more or less what you need for competent autonomous robots, too. If your household robots are orchestrated from your phone, the local security and cloud convenience converge on a single device. No extra servers, etc, reduced cost, all that - local AI is a massive market amplifier.
Let me speculate - we are going in the weird direction of no private property unless you're an overlord that rents his property to peasants. I like to call it the revenge of communism. See how the market behaves in the llm space - it's more viable to share infrastructure than to own it. Imagine the private car revolution in the US was a bus revolution.
Re: A 10 year old Xeon is all you need
#249Hi HN. I wrote this post after getting frustrated by the lack of ways to run the new Gemma 4 Drafter models, and mainstream tools not prioritizing this, and hiding all the performance levers. I ended up getting a modern 26B MoE model (Gemma 4) running at reading speed on an old recycled server with a single Xeon E5-2620 v4 and 128GB of DDR3 RAM (and no GPU). It took a lot of work, but it actually worked out somehow.…
You sure you got DDR3 .. I have 2 e5 v4 rigs at home and both have ddr4 ... Unless I am wrong and 2011-3 supports ddr3 and ddr4
I have a dual e5 v3 that had ddr 4 as well. Been going strong for ten years and still overpowered for what I use it for.
Re: A 10 year old Xeon is all you need
#250We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.
One thing I don’t quite understand:
Wouldn’t it be in Amazon’s interest to run open models and sell time slots at around the cost of running them?
My only guess for why they don’t is that AI labs are currently selling their models at a huge loss, so this isn’t worth Amazon spending low-margin compute on compared to other higher margin products.
What I’m getting at, is maybe we won’t even need to run the models locally for the current status quo to implode. After today’s AI labs run out of free-money runway and actually have to sell their models at a price above running them, there will be the incentive for anyone with compute to just undercut by selling open-models-as-a-service at commodity prices.