Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How oxide cuts data center power consumption in half

oxide.computer

Post reply on HN
Print view View original

11–20 of 185 posts

Re: How oxide cuts data center power consumption in half

#11

Post by AceJohnny2 »

rincebrain wrote: post #7
grecy wrote: post #2

I'm amazed Apple don't have a rack mount version of their M series chips yet. Even for their own internal use in their data centers they'd have to save an absolute boat load on power and cooling given their performance per watt compared to legacy stuff.

I don't think they'd admit much about it even if they had one internally, both because Apple isn't known for their openness about many things, and because they already exited the dedicated server hardware business years ago, so I think they're likely averse to re-entering it without very strong evidence that it would be beneficial for more than a brief period. In particular, while I'd enjoy such a device, Apple's who…

I will add that consumer macOS is a piss-poor server OS.

At one point, for many years, it would just sometimes fail to `exec()` a process. This would manifest as a random failure on our build farm about once/twice a month. (This would manifest as "/bin/sh: fail to exec binary file" because the error type from the kernel would have the libc fall back to trying to run the binary as a script, as normal for a Unix, but it isn't a script)

This is likely stemming from their exiting the server business years ago, and focusing on consumer appeal more than robustness (see various terrible releases, security- and stability-wise).

(I'll grant that macOS has many features that would make it a great server OS, but it's just not polished enough in that direction)

Re: How oxide cuts data center power consumption in half

#12

Post by PreInternet01 »

yjftsjthsd-h wrote: post #9
PreInternet01 wrote: post #5

"If only they used DC from the wall socket, all those H100s would be green" is, not, I think, the hill you want to die on. But, yeah, my three 18MW/y racks agree that more power efficiency would be nice, it's just that Rewrite It In (Safe) Rust is unlikely to help with that...

> it's just that Rewrite It In (Safe) Rust is unlikely to help with that... I didn't see any mention of Rust in the article?

[flagged]

Re: How oxide cuts data center power consumption in half

#13

Post by walrus01 »

They do have a good point here. If you do the total power budget on a typical 1U (discrete chassis, not blade) server which is packed full of a wall of 40mm fans pushing air, the highest speed screaming 40mm 12VDC fans can be 20W electrical load each. It's easy to "spend" at least 120W at maximum heat from the CPUs, in a dual socket system, just on the fans to pull air from the front/cold side of the server through to the rear heat exhaust.

Just going up to 60mm or 80mm standard size DC fans can be a huge efficiency increase in watt-hours spent per cubic meters of air moved per hour.

I am extremely skeptical of the "12x" but using larger fans is more efficient.

from the URL linked:

> Bigger fans = bigger efficiency gains Oxide server sleds are designed to a custom form factor to accommodate larger fans than legacy servers typically use. These fans can move more air more efficiently, cooling the systems using 12x less energy than legacy servers, which each contain as many as 7 fans, which must work much harder to move air over system components.

Re: How oxide cuts data center power consumption in half

#14

Post by unsnap_biceps »

> When we started Oxide, the DC bus bar stood as one of the most glaring differences between the rack-scale machines at the hyperscalers and the rack-and-stack servers that the rest of the market was stuck with. That a relatively simple piece of copper was unavailable to commercial buyers

It seems that 0xide was founded in 2019 and Open Compute Project had been specifying dc bus bars for 6 years at that point. People could purchase racks if they wanted, but it seems like, by large, people didn't care enough to go whole hog in on it.

Wonder if the economics have changed or if it's still just neat but won't move the needle.

Re: How oxide cuts data center power consumption in half

#15

Post by AceJohnny2 »

AceJohnny2 wrote: post #11
rincebrain wrote: post #7

Earlier quoted context omitted.

I don't think they'd admit much about it even if they had one internally, both because Apple isn't known for their openness about many things, and because they already exited the dedicated server hardware business years ago, so I think they're likely averse to re-entering it without very strong evidence that it would be beneficial for more than a brief period. In particular, while I'd enjoy such a device, Apple's who…

I will add that consumer macOS is a piss-poor server OS. At one point, for many years, it would just sometimes fail to `exec()` a process. This would manifest as a random failure on our build farm about once/twice a month. (This would manifest as "/bin/sh: fail to exec binary file" because the error type from the kernel would have the libc fall back to trying to run the binary as a script, as normal for a Unix, but i…

> as normal for a Unix

veering offtopic, did you know macOS is a certified Unix?

https://www.opengroup.org/openbrand/register/brand3581.htm

As I recall, Apple advertised macOS as a Unix without such certification, got sued, and then scrambled to implement the required features to get certification as a result. Here's the story as told by the lead engineer of the project:

https://www.quora.com/What-goes-into-making-an-OS-to-be-Unix...

Re: How oxide cuts data center power consumption in half

#16

Post by walrus01 »

grecy wrote: post #2

I'm amazed Apple don't have a rack mount version of their M series chips yet. Even for their own internal use in their data centers they'd have to save an absolute boat load on power and cooling given their performance per watt compared to legacy stuff.

Companies buying massive cloud scale server hardware want to be able to choose from a dozen different Taiwanese motherboard manufacturers. Apple is in no way motivated to release or sell the M3/M4 CPUs as a product that major east asia motherboard manufacturers can design their own platform for. Apple is highly invested in tightly integrated ecosystems where everything is soldered down together in one package as a consumer product (take a look at a macbook air or pro motherboard for instance).

Re: How oxide cuts data center power consumption in half

#17

Post by shivak »

> > The power shelf distributes DC power up and down the rack via a bus bar. This eliminates the 70 total AC power supplies found in an equivalent legacy server rack within 32 servers, two top-of-rack switches, and one out-of-band switch, each with two AC power supplies

This creates a single point of failure, trading robustness for efficiency. There's nothing wrong with that, but software/ops might have to accommodate by making the opposite tradeoff. In general, the cost savings advertised by cloud infrastructure should be more holistic.

Re: How oxide cuts data center power consumption in half

#18

Post by sophacles »

PreInternet01 wrote: post #12
yjftsjthsd-h wrote: post #9

Earlier quoted context omitted.

> it's just that Rewrite It In (Safe) Rust is unlikely to help with that... I didn't see any mention of Rust in the article?

[flagged]

That's an interesting take. What's your reasoning? Whats your evidence?

Re: How oxide cuts data center power consumption in half

#19

Post by walrus01 »

shivak wrote: post #17

> > The power shelf distributes DC power up and down the rack via a bus bar. This eliminates the 70 total AC power supplies found in an equivalent legacy server rack within 32 servers, two top-of-rack switches, and one out-of-band switch, each with two AC power supplies This creates a single point of failure, trading robustness for efficiency. There's nothing wrong with that, but software/ops might have to accommodat…

The whole thing with eliminating 70 discrete 1U server size AC-to-DC power supplies is nothing new. It's the same general concept as the power distribution unit in the center of an open compute platform rack design from 10+ years ago.

Everyone who's doing serious datacenter stuff at scale knows that one of the absolute least efficient, labor intensive and cabling intensive/annoying ways of powering stuff is to have something like a 42U cabinet with 36 servers in it, each of them with dual power supplies, with power leads going to a pair of 208V 30A vertical PDUs in the rear of the cabinet. It gets ugly fast in terms of efficiency.

The single point of failure isn't really a problem as long as the software is architected to be tolerant of the disappearance of an entire node (mapping to a single motherboard that is a single or dual cpu socket config with a ton of DDR4 on it).

Re: How oxide cuts data center power consumption in half

#20

Post by walrus01 »

unsnap_biceps wrote: post #14

> When we started Oxide, the DC bus bar stood as one of the most glaring differences between the rack-scale machines at the hyperscalers and the rack-and-stack servers that the rest of the market was stuck with. That a relatively simple piece of copper was unavailable to commercial buyers It seems that 0xide was founded in 2019 and Open Compute Project had been specifying dc bus bars for 6 years at that point. People…

Things like -48VDC bus bars in the 'telco' world significantly predate the OCP, all the way back to like 1952 in the Bell system.

In general, the telco world concept hasn't changed much. You have AC grid power coming from your local utility into some BIG ASS RECTIFIERS which create -48VDC (and are responsible for charging your BIG ASS BATTERY BANK to float voltage), then various DC fuses/breakers going to distribution of -48VDC bus bars powering the equipment in a CO.

Re: Open Compute, the general concept of what they did was go to a bunch of 1U/2U server power supply manufacturers and get them to make a series of 48VDC-to-12VDC power supplies (which can be 92%+ efficient), and cut out the need for legacy 5VDC feed from power supply into ATX-derived-design x86-64 motherboards.

Post reply on HN