Live data from Hacker News

Facebook announces next-generation Open Rack frame

code.fb.com

31–40 of 46 posts

Re: Facebook announces next-generation Open Rack frame

#31
post #13

Earlier quoted context omitted.

Facebook does use it. They design their own hardware, but they are do not manufacture it themselves, they aren't in the hardware manufacturing business. They parter with companies like Quanta to do that. Here's an older article that explains the relationship: https://venturebeat.com/2014/01/29/facebook-quanta/

I'm well aware that. I'm saying that in a sense that they don't buy or use much of the original "ocp platform" they show off at their events. And much of it was said to have ended after limited deployment in their Prineville DC, after which they switched back to regular OEM quanta gear with just few things like blue-green handlebars added, and "barebone" motherboard trays. I've been whispered that the biggest buyers…

Facebook's entire blob storage and data warehouse (multi-exabyte) is run completely on OCP storage hardware built by ODMs, of which Quanta is included. Anyone telling you that we don't use OCP is grossly misinformed.

Source: I was on the blob storage team when we migrated all of our data from OCP's gen1 storage design [1] to the new gen2 storage design [2].

1: https://www.opencompute.org/documents/facebook-open-vault-st...

2: https://www.opencompute.org/documents/facebook-bryce-canyon-...

edit: corrected links.

Re: Facebook announces next-generation Open Rack frame

#32
post #5

Earlier quoted context omitted.

Makes sense but has nice second order effects. If we could standardise more we can optimise for power efficiency, things like more DC in DCs

FIY, the original telecom rack equipment been standardised on 48V since time immemorial. So, at 48V you can simply use off the shelf telecom power supplies

I've been trying to find cheap 48v power supplies for 1A and less telcom gear, but they seem to be pretty spendy. Cheaper to buy old gear that includes the PSU than order the PSU itself.

Re: Facebook announces next-generation Open Rack frame

#33
post #22

Earlier quoted context omitted.

DC power wise, the original reason why they did this is that it's grossly inefficient to have a 42RU worth of 1U servers, each with its own individual 110-240VAC to 12/5VDC power supply in it. Or the equivalent of that with four-servers-in-2RU type setups, but also with AC to DC power supplies. They centralized the AC to DC conversion in a single unit in the rack, feeding either 277VAC or 480VAC to each rack, and ran…

There are 90%+ efficient reasonably cheap flex psus, but the mainstream is around 80%. So, you would not be saving much if you only have access to 110 or 220 mains. The economy changes though if you can get access to industrial 3 phase linkup at 400V, and can avoid additional low step down transformers.

> So, you would not be saving much if you only have access to 110 or 220 mains.

"Much" is a relative term here, no? Like, doesn't that kind of depend at the scale at which Facebook operates?

Re: Facebook announces next-generation Open Rack frame

#34
post #22

Earlier quoted context omitted.

DC power wise, the original reason why they did this is that it's grossly inefficient to have a 42RU worth of 1U servers, each with its own individual 110-240VAC to 12/5VDC power supply in it. Or the equivalent of that with four-servers-in-2RU type setups, but also with AC to DC power supplies. They centralized the AC to DC conversion in a single unit in the rack, feeding either 277VAC or 480VAC to each rack, and ran…

There are 90%+ efficient reasonably cheap flex psus, but the mainstream is around 80%. So, you would not be saving much if you only have access to 110 or 220 mains. The economy changes though if you can get access to industrial 3 phase linkup at 400V, and can avoid additional low step down transformers.

I read somewhere that part of it was also doing the short-term battery backup on a rack-level, without needing conversion back to mains AC and then again back to DC in the server for it.

Re: Facebook announces next-generation Open Rack frame

#35
post #22

Earlier quoted context omitted.

DC power wise, the original reason why they did this is that it's grossly inefficient to have a 42RU worth of 1U servers, each with its own individual 110-240VAC to 12/5VDC power supply in it. Or the equivalent of that with four-servers-in-2RU type setups, but also with AC to DC power supplies. They centralized the AC to DC conversion in a single unit in the rack, feeding either 277VAC or 480VAC to each rack, and ran…

There are 90%+ efficient reasonably cheap flex psus, but the mainstream is around 80%. So, you would not be saving much if you only have access to 110 or 220 mains. The economy changes though if you can get access to industrial 3 phase linkup at 400V, and can avoid additional low step down transformers.

You can use microwave oven transformer to step to 1000VDC then simply use step down converter in each rack.

I am doing this for my home lab with 100 servers.

Re: Facebook announces next-generation Open Rack frame

#36
post #13

Earlier quoted context omitted.

I'm well aware that. I'm saying that in a sense that they don't buy or use much of the original "ocp platform" they show off at their events. And much of it was said to have ended after limited deployment in their Prineville DC, after which they switched back to regular OEM quanta gear with just few things like blue-green handlebars added, and "barebone" motherboard trays. I've been whispered that the biggest buyers…

Facebook's entire blob storage and data warehouse (multi-exabyte) is run completely on OCP storage hardware built by ODMs, of which Quanta is included. Anyone telling you that we don't use OCP is grossly misinformed. Source: I was on the blob storage team when we migrated all of our data from OCP's gen1 storage design [1] to the new gen2 storage design [2]. 1: https://www.opencompute.org/documents/facebook-open-vault…

Well, then it is.

What do you use for regular servers these day?

Re: Facebook announces next-generation Open Rack frame

#37
post #12

Could ocp make a rack that behaves like a blade server? Construct a server rack that works like a network Patch panel and server power bus? Thus one would just install the server in the rack slot, no more Cables to the server. Bonus award if a robot could slide in the server in the rack.

What’s the current industry practice for blades? Does AWS GCP, Azure use them?

> Does AWS GCP, Azure use them?

No. Cost / value is not in alignment there. You want your servers to be cheap and disposable.

Plus lose the control plane and you lose all the blades within it. There's just no value to it unless you're doing workloads that benefit from a high degree of locality. Cloud services focus on having their networks be as fast as possible so as to reduce the disadvantage of not being highly local.

Oracle (my employer), and AWS have been announcing various HPC cloud products where we're starting to focus on highly local, servers with fast interconnects, and it's still not looking at blades. HPC workloads are rather untapped by clouds so far, and it's a big market.

Re: Facebook announces next-generation Open Rack frame

#38
post #20
post #19

Earlier quoted context omitted.

So your theory is, what, that Facebook spends tons of money designing multiple generations of hardware that they don't use? Why would that make any sense?

It doesn't and that is what baffles me

So.... maybe it's worth considering that your initial premise is faulty?

Re: Facebook announces next-generation Open Rack frame

#39
post #21
post #9

Earlier quoted context omitted.

without actually manufacturing or using anything of it. As I know, facebook still buys very plain OEM servers from Quanta This is false, as you can see from publicly available pictures of FB datacenters. Facebook purchases OCP servers from ODMs.

ODMs or OEMs?

ODMs = System Integrators, if you're familiar with that term instead?

There's several companies that will do large scale bespoke-design server manufacturing and assembly work.

Amazon, Facebook, Microsoft et al. all order servers through them that are built to specifications their hardware engineers have designed (usually in collaboration with them). Once you get above a certain scale, the value proposition of OEMs goes out the window.

Re: Facebook announces next-generation Open Rack frame

#40
post #7

The sooner this centralized model of storing data ends, the better.

What does that have to do with an open specification for racks of computers? This is a design that anyone could potentially adopt, and gain from the engineering efforts that those involved in OCP have done.

There's even a marketplace you can purchase Open Rack components through (and you could likely also go more direct to those companies rather than via OCP's marketplace): https://www.opencompute.org/products

Post reply on HN