Servers as they should be – shipping early 2022 • Hacker News Board

Re: Servers as they should be – shipping early 2022

#121

Post by zozbot234 » Wed, May 26, 2021, 8:59 PM UTC

yabones wrote: post #108

Aesthetically, absolutely love it. In reality I would never want this type of hardware... It reminds me of the old boat anchor bladecenter rigs we used to use. They were great, up until you had to replace one of the blades after the support was up. It's not always practical to replace hardware every 3 years like we're supposed to, so this type of stuff sticks around and gets some barnacles. What would be fantastic wo…

> What would be fantastic would be if the entire industry committed to an open spec for large chassis like this with a standardized networking and storage overlay

Isn't the Open Compute Project supposed to be working on that kind of stuff?

Re: Servers as they should be – shipping early 2022

#122

Post by tw04 » Wed, May 26, 2021, 8:59 PM UTC

bcantrill wrote: post #110

atonse wrote: post #9

This looks interesting (although I'm not in the target market, too small)... But if I were looking at this, judging from the quality of people they've amassed in their engineering team, is there any chance they won't be acquired in 6 months? To anyone looking to take a bet on this, what is the answer to "what's your plan for when your stellar team gets acquired?" And what answer will satisfy that buyer? Update: Addin…

Hi! So, at every step -- from conception to funding to building the team and now building the product -- we have done so to build a big, successful public company. Not only do we (the founders) share that conviction, but it is shared by our investors and employees as well. For better or for ill, we are -- as we memorably declared to one investor -- ride or die. Also, if it's of any solace, I really don't think any of…

Isn't that what some of your old Sun compatriots thought with DSSD? :)

Congrats on the announcement, here's hoping you're right! This looks too interesting to be swallowed by Oracle or HPe.

Re: Servers as they should be – shipping early 2022

#123

Post by Tuna-Fish » Wed, May 26, 2021, 9:02 PM UTC

kaliszad wrote: post #69

_delirium wrote: post #48

Earlier quoted context omitted.

The density they're getting here is significantly higher than AWS Outposts, which is interesting. The top-end (~$600k) AWS Outposts seem to max out at around 1k CPUs and 4.5 TB RAM in a rack (e.g. 12x m5.24xlarge = 12x 384 GB), while this rack can house 2k CPUs and 30 TB (!) RAM.

Yes, well it isn't that dense either. As I have written, it's 32 CPUs (16x 2 CPUs). 1 TB of RAM per CPU is not that huge a deal, it's perhaps 16x 64 GB (Milan uses 8 channels, 2 DIMMs per channel is reasonable), if you consider that is 16 GB of RAM per core. In HPC, you would probably shrink it to 1/4 of the volume (half width, 1 U dual socket server). Oxide probably focuses on optimal thermal efficiency since their…

There are 32 half-width compute blades there, so probably single socket servers with 64 cores each.

Re: Servers as they should be – shipping early 2022

#124

Post by notacoward » Wed, May 26, 2021, 9:02 PM UTC

dralley wrote: post #98

jmartrican wrote: post #87

Why the elevation constraint? "The elevation of the room where the rack is installed must be below 10,005 feet (3,050 meters)."

At 10,000 the air pressure is 1/3 less than at sea level, less density means less capacity for carrying heat, so the cooling might not be sufficient.

Bingo. I've personally had to deal with this in other high-density systems. Less cooling not only has the obvious effects, but also reduces PS efficiency which can cause other problems. Cosmic-ray-induced memory errors can also be a problem at those altitudes (or even half that). That's a bit easier to deal with in principle, but the rate of ECC scrubbing required can start to impact performance. Stack that on top of thermal CPU throttling, and you'll have a system that's just slower than it should be. Just as importantly, the slowdown will be uneven across components, so it's effectively a different kind of system when you're debugging low-level issues.

I think it's a good sign that they're aware of the additional support issues associated with higher altitude. Shows that they've really thought things through.

Re: Servers as they should be – shipping early 2022

#125

Post by zozbot234 » Wed, May 26, 2021, 9:05 PM UTC

dsr_ wrote: post #119

StreamBright wrote: post #78

Earlier quoted context omitted.

I thought M1 had the best price/performance numbers, too bad Apple does not sell CPUs.

Or, more relevant to this discussion, servers.

They sell rack-mountable hardware (Mac Pros) running a certified Unix OS. It's only a matter of time until those are based on Apple Silicon too.

Re: Servers as they should be – shipping early 2022

#126

Post by ex_amazon_sde » Wed, May 26, 2021, 9:05 PM UTC

_jal wrote: post #57

It sure looks pretty, but appears to be - - dedicated to virtualization, done their way - rather inflexible in hardware specs - vendor-locked at the rack - if you have hardware from someone else, it can't live in the same cabinet I guess if you just want a pretty data center in a box and look like what they consider a 'normal' enterprise to be, it might appeal. But I'm not sure how many people asked for Apple-style h…

Vendor lock-in at rack level and custom servers? Sounds like blade servers.

A smart company would stay away from this kind of strong lock-in.

Re: Servers as they should be – shipping early 2022

#127

Post by samstave » Wed, May 26, 2021, 9:06 PM UTC

boulos wrote: post #47

richardwhiuk wrote: post #27

Earlier quoted context omitted.

Suspect each node is both storage and compute. Guessing they aren't counting threads (they say "cores"), so 64 cores per socket, 128 cores per server, 16 servers => 2048 cores.

Duh! I got tricked by the things near the PDU as "oh, these must be the pure-compute nodes". So maybe that's the better question: what are the 4U worth of stuff surrounding the power? More networking stuff? Management stuff? (There was some swivel to the back of the rack / with networking, but I can't find it now) Edit: Ahh! The rotating view is on /product and so that ~4U is the fiber. (Hat tip to Jon Olson, too)

Control-plane most likely, and having a mid-centered PDU probably adds to heat on the upper stack, which shortens life over time.

As someone who has designed quite a few datacenters, whats more interesting to me in this evolution of computing is the reduction in cabling.

Cabling in a DC is a huge suck on all aspects - plastics, power, blah blah blah - the list is long....

But there are a LOT of cabling companies that do LV out there - so the point is that when these types of systems get more "obelisk" like, are many of these companies going to die? (I'm looking at you Cray and SGI.)

When I worked at Intel - I had a friend who was a proc designer at MIPS - and we talked about rack insertion and a global back-plane for the rack (which we all know to be common now) - but this was ~1997 or so... but when I built the Brocade HQ - cables were still massive and it was an art to properly dress them.

Lucas was the same - so many human work hours spent on just cable mgmt...

Their diagrams of system resiliency is odd in my opinion:

https://i.imgur.com/GB0fzIl.png

That looks like a ton of failures that they can negotiate...

Whats weird is the SPF isn't going to be in your DC/HQ/Whatever - its going to be outside - this is why we have always sought +2+ carrier ISPs or built private infra...

A freaking semi truck crashed into a telephone pole in Sacramento the other day and wiped comcast off the map to half the region.

https://sacramento.cbslocal.com/2021/05/25/citrus-heights-an...

Thats ONE fiber line that brought down 100K+ connections...

---

EDIT: I guess what I am actually saying is that this entire marketing strat is to convince any companies that *"failure is imminent and please buy things that are going to fail, but don't worry because you bought plenty more things to live beyond the epic failure that these devices will have"*

---

Not to discredit anything this company has going for its product - but their name is literally "RUST" (*oxide*) --- which we all know is what kills metal.

And what do we call servers: *Bare Metal*

Re: Servers as they should be – shipping early 2022

#128

Post by bcantrill » Wed, May 26, 2021, 9:07 PM UTC

tw04 wrote: post #122

bcantrill wrote: post #110

Earlier quoted context omitted.

Hi! So, at every step -- from conception to funding to building the team and now building the product -- we have done so to build a big, successful public company. Not only do we (the founders) share that conviction, but it is shared by our investors and employees as well. For better or for ill, we are -- as we memorably declared to one investor -- ride or die. Also, if it's of any solace, I really don't think any of…

Isn't that what some of your old Sun compatriots thought with DSSD? :) Congrats on the announcement, here's hoping you're right! This looks too interesting to be swallowed by Oracle or HPe.

Fortunately, some of those same DSSD folks have joined us at Oxide -- and let's just say that they are of like mind with respect to Oxide's approach. ;)

Re: Servers as they should be – shipping early 2022

#129

Post by [deleted] » Wed, May 26, 2021, 9:08 PM UTC

bcantrill wrote: post #110

atonse wrote: post #9

This looks interesting (although I'm not in the target market, too small)... But if I were looking at this, judging from the quality of people they've amassed in their engineering team, is there any chance they won't be acquired in 6 months? To anyone looking to take a bet on this, what is the answer to "what's your plan for when your stellar team gets acquired?" And what answer will satisfy that buyer? Update: Addin…

Hi! So, at every step -- from conception to funding to building the team and now building the product -- we have done so to build a big, successful public company. Not only do we (the founders) share that conviction, but it is shared by our investors and employees as well. For better or for ill, we are -- as we memorably declared to one investor -- ride or die. Also, if it's of any solace, I really don't think any of…

[deleted]

Re: Servers as they should be – shipping early 2022

#130

Post by qbasic_forever » Wed, May 26, 2021, 9:09 PM UTC

jeffbee wrote: post #13

caeril wrote: post #4

Earlier quoted context omitted.

Cantrill has always said it's for people who want Facebook class on-premise infrastructure but don't have a $900B market cap and a hundred engineers designing and building custom boxes. Oh, and open firmware.

Can definitely see it for a company size of Dropbox, big enough to already be working with ODMs, big enough to be sensitive to the kind of headaches you get from a heterogeneous fleet of ILOM processors designed by deranged engineers.

Dropbox is big enough that they could just acquire Oxide now before they even get to market. That might even be the plan all along. I can't imagine there are more than like a dozen companies that are their target market, i.e. big enough to need Facebook-level datacenters but not big enough (yet) to have that engineering team.