Live data from Hacker News

Servers as they should be – shipping early 2022

oxide.computer

101–110 of 264 posts

Re: Servers as they should be – shipping early 2022

#101
The storage is always the difficult part in these architectures. Are you distributing across all nodes? It appears that each sled is an individual compute unit with 10 drives. Are the drives on a proverbial island and only accessible to that local node, or is there some distributed storage going on that you can talk about?

On paper with RDMA and NVMe-OF you could access any drive from any compute unit... but that's easier said than done :)

Re: Servers as they should be – shipping early 2022

#102
post #5
post #2

Who is the target customer for this?

This level of integration surely won't come cheap, from what I recall of Server purchasing a target price of ~200-500k per rack would be expected on a TCO of roughly 2x the rack price over 3 years. (Assuming you are buying from Quanta/SuperMicro or other commodity integrator) It's possible the prices are different now, but you would need customers looking to drop > 1 million dollars in CapEx for the management capabi…

They'd have to sell these at a significant loss to make up for the risk any company would have to take to build out a DC on first generation hardware from a startup.

Re: Servers as they should be – shipping early 2022

#103

Earlier quoted context omitted.

The density they're getting here is significantly higher than AWS Outposts, which is interesting. The top-end (~$600k) AWS Outposts seem to max out at around 1k CPUs and 4.5 TB RAM in a rack (e.g. 12x m5.24xlarge = 12x 384 GB), while this rack can house 2k CPUs and 30 TB (!) RAM.

Yes, well it isn't that dense either. As I have written, it's 32 CPUs (16x 2 CPUs). 1 TB of RAM per CPU is not that huge a deal, it's perhaps 16x 64 GB (Milan uses 8 channels, 2 DIMMs per channel is reasonable), if you consider that is 16 GB of RAM per core. In HPC, you would probably shrink it to 1/4 of the volume (half width, 1 U dual socket server). Oxide probably focuses on optimal thermal efficiency since their…

The problem with pushing higher compute density is you're running into the limits of what most DCs can provide in terms of power and cooling for a single rack. Usually it's specialized HPC facilities or hyperscalers pushing the power and cooling to handle stuff like that. Those people aren't likely Oxide's customers - they've already got their own hardware solutions.

Re: Servers as they should be – shipping early 2022

#104
post #71
post #66

Earlier quoted context omitted.

Oxide makes the hardware, so it makes sense to use them as the root of trust since you already have to trust them to not make backdoored hardware. Why bother adding more parties? Also, for remote attestation to make sense, it needs to be done in the hardware itself (ie. keys burned into the silicon). I'm not sure how that's supposed to work if you add your own keys, or whether that would even make sense.

"it needs to be done in the hardware itself (ie. keys burned into the silicon)" - this isn't true; this is confusing Trusted Boot and Secure Boot, which are not the same thing (nor is it the only way of implementing Secure Boot). Owner-controlled remote attestation is entirely viable, e.g. Talos II is capable of this with a FlexVer module.

> "it needs to be done in the hardware itself (ie. keys burned into the silicon)" - this isn't true; this is confusing Trusted Boot and Secure Boot, which are not the same thing (nor is it the only way of implementing Secure Boot).

I meant as opposed to keys/signing done in software.

>Owner-controlled remote attestation is entirely viable, e.g. Talos II is capable of this with a FlexVer module.

I skimmed the product brief[1] and it looks like it's basically a TPM that has a secure communications channel (as opposed to LPC which can be MITMed)? I'm not really sure how this is an improvement, because you're still relying on the hardware vendor to send the PCR values. So at the end of the day you still have to trust the hardware vendor, although the signing is done by you, but I'm not really sure how this adds any benefit.

[1] https://www.raptorengineering.com/TALOS/documentation/flexve...

Re: Servers as they should be – shipping early 2022

#106

Looks like a white labeled viking VDS2249R running custom software.

This is not my area of expertise, but it does look like that[0]. That "custom software," though, is where the magic often lies. As a software person that worked at hardware companies for most of my career, I know all too well, how disrespectful hardware people are of software. If they have a good software-respecting management chain, then it might be pretty awesome. [0] https://www.prnewswire.com/news-releases/viking…

Well if Oxide's software stack is going to be open source, I guess we'll get a good look into their secret sauce.

Re: Servers as they should be – shipping early 2022

#108
Aesthetically, absolutely love it.

In reality I would never want this type of hardware... It reminds me of the old boat anchor bladecenter rigs we used to use. They were great, up until you had to replace one of the blades after the support was up. It's not always practical to replace hardware every 3 years like we're supposed to, so this type of stuff sticks around and gets some barnacles.

What would be fantastic would be if the entire industry committed to an open spec for large chassis like this with a standardized networking and storage overlay... But that would never happen because vendor lock-in is the big money maker in 'enterprise'.

But wow, absolutely gorgeous machines.

Re: Servers as they should be – shipping early 2022

#109
It's interesting that the RAM/CPU ratio is about double the default shapes from AWS/GCP. In practice I have generally seen those shapes run on the low side of CPU utilization for most workloads, so I think the choice makes sense.

I'm curious if ARC will be running with primarycache=metadata to rely on low latency storage and in-VM cache, otherwise I could see ARC using a fair bit of that RAM overhead in the hosts.

Re: Servers as they should be – shipping early 2022

#110
post #9

This looks interesting (although I'm not in the target market, too small)... But if I were looking at this, judging from the quality of people they've amassed in their engineering team, is there any chance they won't be acquired in 6 months? To anyone looking to take a bet on this, what is the answer to "what's your plan for when your stellar team gets acquired?" And what answer will satisfy that buyer? Update: Addin…

Hi! So, at every step -- from conception to funding to building the team and now building the product -- we have done so to build a big, successful public company. Not only do we (the founders) share that conviction, but it is shared by our investors and employees as well. For better or for ill, we are -- as we memorably declared to one investor -- ride or die.

Also, if it's of any solace, I really don't think any of the existing players would be terribly interested in buying a company that has so thoroughly and unequivocally rejected so many of their accrued decisions! ;) I'm pretty sure their due diligence would reveal that we have taken a first principles approach here that is anathema to the iterative one they have taken for decades -- and indeed, these companies have shown time and time again that they don't want to risk their existing product lines to a fresh approach, no matter how badly customers want it.

Post reply on HN