When you use page up and page down key when reading that blog the first line on the screen is obscured by the floating bar or what ever it is. It is not even needed for reading.
A 10 year old Xeon is all you need
61–70 of 301 posts
Re: A 10 year old Xeon is all you need
#62Nice post and technically impressive work. I agree we need to understand the build pipeline and be able to do things locally. However, depending on your electricity cost, it might not make sense financially. These old servers are not energy efficient at all (I'm guessing that old Xeon server will easily pull 200W on load), and that model is currently at 0.1$/0.3$ per 1M tokens (with 76 tps and 262k context) in Openro…
It should be closer to 85W on load. And it's incredibly silent on even a low end cooler. I rarely get above 50° Celcius.
Only when you remove it from the original server or enable low fan mode (if available). Most 1U/2U cases will happily blow at full speed well over 90db.
You likely need to replace the flow-through server chassis system with an active "normal" cooler to achieve a bit of silence.
85W might be about right. My old server CPU is in the same ballpark and compiling kernels it reached about 90w in power usage. If you want to keep it running: idle is not very low power unless you have one of the "low power" L versions, keep that in mind.
Re: A 10 year old Xeon is all you need
#63Nice post and technically impressive work. I agree we need to understand the build pipeline and be able to do things locally. However, depending on your electricity cost, it might not make sense financially. These old servers are not energy efficient at all (I'm guessing that old Xeon server will easily pull 200W on load), and that model is currently at 0.1$/0.3$ per 1M tokens (with 76 tps and 262k context) in Openro…
2620v4 is not a power slurping beast. Depending on the server board, it might not be either. Servers are often loud, but it depends.
There's a lot of budget hosting built around chips like these, and they're suprisingly power efficient.
Re: A 10 year old Xeon is all you need
#64I think one overlooked advantage of older Xeon systems is their availability. Many people can experiment with local AI deployments at a fraction of the cost of building a brand-new setup.
Re: A 10 year old Xeon is all you need
#65The E5-2620 v4 is great. Have been using it for 10 years now. Wanted to upgrade until I saw current prices. I have 64 GB ddr4. Paired it with rx 9060 xt 16 GB and games run as fast as ever. Perhaps the cpu is a slight bottleneck in DOOM The Dark Ages, but i'm at 60 fps, so no problem. Light llm on the gpu is a nobrainer, and it's cool to see that things can be tuned to run ok on the cpu. I bought 2667 v4 a month ago…
> The E5-2620 v4 is great. Have been using it for 10 years now.
Re: A 10 year old Xeon is all you need
#66Earlier quoted context omitted.
It should be closer to 85W on load. And it's incredibly silent on even a low end cooler. I rarely get above 50° Celcius.
OK, then you're in luck. I had a bunch of old 1U rack servers and even in the next room it was too annoying to run them (they had a bunch of 40mm fans which always ran at full speed, because in a server room, no one can hear you scream).
Could it just be really bad cooling? Looking at 9800X3D, it seems like it's running in a similar range wrt TDP unless you really push the 9800X3D. I'm comparing with desktop cpu's because that's what my workload is. cpu governor is set to performance (no schedutil). No audible change in fan speed during heavy compilation or gaming (very silent humming), and i don't have any fans beside cheap intake, cpu and exhaust fans (1 each) + an excessive amount of dust.
Re: A 10 year old Xeon is all you need
#67Earlier quoted context omitted.
It should be closer to 85W on load. And it's incredibly silent on even a low end cooler. I rarely get above 50° Celcius.
Only when you remove it from the original server or enable low fan mode (if available). Most 1U/2U cases will happily blow at full speed well over 90db. You likely need to replace the flow-through server chassis system with an active "normal" cooler to achieve a bit of silence. 85W might be about right. My old server CPU is in the same ballpark and compiling kernels it reached about 90w in power usage. If you want to…
Get a 4U case, many options if you want to combine it with a NAS. Not hard to cool and keep somewhat quiet. If you can store it in a closet or something that helps too.
Well, you can use it for lots of other things as well.
Compared to the cloud you can probably save up to buy a new server every month. And don't underestimate the gains of having something to experiment on and play with.
Re: A 10 year old Xeon is all you need
#68Earlier quoted context omitted.
OK, then you're in luck. I had a bunch of old 1U rack servers and even in the next room it was too annoying to run them (they had a bunch of 40mm fans which always ran at full speed, because in a server room, no one can hear you scream).
Could it just be really bad cooling? Looking at 9800X3D, it seems like it's running in a similar range wrt TDP unless you really push the 9800X3D. I'm comparing with desktop cpu's because that's what my workload is. cpu governor is set to performance (no schedutil). No audible change in fan speed during heavy compilation or gaming (very silent humming), and i don't have any fans beside cheap intake, cpu and exhaust f…
These servers had no fan control whatsoever, they always ran full blast. That's not untypical for rack servers, because as written: they are designed for server rooms, and you're supposed to wear ear protection there anyway... Yes, I could've modified them, but I ditched them because running them simply made no sense (especially the high idle power consumption was ridiculous).
Re: A 10 year old Xeon is all you need
#69I may have missed this in the article, but:
What was the net effect of the optimisations? How much faster did it get?
Re: A 10 year old Xeon is all you need
#70Now we need someone try run Kimi K2.6 on old Xeon and DDR3. After all these platforms do support up to 768GB RAM.
It’ll work but yield a token per minute. With ancient servers the throughput is the limiting aspect not mem size