Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Servers as they should be – shipping early 2022

oxide.computer

Post reply on HN
Print view View original

41–50 of 264 posts

Re: Servers as they should be – shipping early 2022

#41

Post by hlandau »

gruez wrote: post #37
hlandau wrote: post #22

"Attests the software version is the version that is valid and shipped by the Oxide Computer Company" So in other words these servers will implement restrictive code signing practices and will be vendor-controlled, not owner-controlled? This is not my idea of "secure", and really in the wake of things like the Solarwinds or RSA hacks it shouldn't be anyone's idea of secure. Vendor-holds-the-keys is not an acceptable…

"Attest" refers to https://en.wikipedia.org/wiki/Trusted_Computing#REMOTE-ATTES... , not forced code signing

I'm familiar with the concept. Does this mean that attestation to a different root of trust than Oxide will also be feasible, and that this is just a default?

Re: Servers as they should be – shipping early 2022

#42

Post by kaliszad »

"Only" 2048 CPU cores per rack is actually not that much by nowadays standards - its 16 U of 2x 64 core CPUs. Perhaps is could be more U if they used the lower core counts but e.g. higher frequency per core SKUs but I don't think they do. (And the picture kind of confirms it). They use 2U servers though so they are able to use lower speed but bigger fans and have more expansion cards and 2,5" form factor drives perhaps. The of course have to fit storage, which needs lots of CPU PCIe lanes for all the NVMe storage and networking (probably 2 or 4 U) and power conversion to power the bus bar and more somewhere. They probably use the 42 U+ standard 19" racks to fit in standard customers DCs. They also don't have such a high power budget as custom DCs for cloud providers do.

1 PB of flash is quite a bit but you could get perhaps 5x as much with HDDs probably (even with a relatively low density of 40x 12 x 12 TB). The problem really is I think, they wouldn't be able to write the HDD firmware in Rust in time (or at all, because no HDD manufacturer would sell an HDD to them without making sure their proprietary firmware is used). SSDs don't necessarily have this property as they are much more like the other components of a modern server.

Re: Servers as they should be – shipping early 2022

#43

Post by tediousdemise »

stefan_ wrote: post #19

Why are we hard coupling the hardware to the software? The whole secret of the success of M1 and ARM in servers is that lots of software has long ago stopped being hyper-aware of what hardware it is running on. What software are we talking about anyways? It's all incredibly vague, but it seems to reach all the way into the Kubernetes sphere. Why would I run this over something I can use on my next job?

> Why are we hard coupling the hardware to the software? The whole secret of the success of M1 and ARM in servers is that lots of software has long ago stopped being hyper-aware of what hardware it is running on.

The software running on M1 is a bespoke fit for it. That's why the performance in macOS on M1 is phenomenal. It was custom made to execute optimally on it.

Re: Servers as they should be – shipping early 2022

#44

Post by rcxdude »

stefan_ wrote: post #19

Why are we hard coupling the hardware to the software? The whole secret of the success of M1 and ARM in servers is that lots of software has long ago stopped being hyper-aware of what hardware it is running on. What software are we talking about anyways? It's all incredibly vague, but it seems to reach all the way into the Kubernetes sphere. Why would I run this over something I can use on my next job?

It's basically a mini-ec2 in your server room. The software and hardware give you a platform to deploy VMs configured however you want.

Re: Servers as they should be – shipping early 2022

#46

Post by kaliszad »

ptomato wrote: post #31
fsociety wrote: post #3

I kind of want one of these in my home, I wonder what the price range will be.

for what they're advertising there, on the order of $1mm, based on underlying hardware costs.

Actually, standard Azure rack is on the order of $1.1 mm of hardware depending on SKU if I am not mistaken. So I would guess, it could be more like $2 mm. There is the aspect of management and other vendors Dell/EMC + VMware like you to pay way more than the hardware costs for e.g. VXRail/ vSphere licences. That is the real target.

Re: Servers as they should be – shipping early 2022

#47

Post by boulos »

richardwhiuk wrote: post #27
boulos wrote: post #21

First, congrats! But second, I'd love to understand the compute vs storage tradeoff chosen here. Looking at the (pretty!) picture [1], I was shocked to see "Wow, it's mostly storage?". Is that from going all flash? Heading to https://oxide.computer/product for more details, lists: - 2048 cores - 30 TB of memory - 1024 TB of flash (1 PiB) Given how much of the rack is storage, I'm not sure which Milan was chosen (and…

Suspect each node is both storage and compute. Guessing they aren't counting threads (they say "cores"), so 64 cores per socket, 128 cores per server, 16 servers => 2048 cores.

Duh! I got tricked by the things near the PDU as "oh, these must be the pure-compute nodes".

So maybe that's the better question: what are the 4U worth of stuff surrounding the power? More networking stuff? Management stuff? (There was some swivel to the back of the rack / with networking, but I can't find it now)

Edit: Ahh! The rotating view is on /product and so that ~4U is the fiber. (Hat tip to Jon Olson, too)

Re: Servers as they should be – shipping early 2022

#48

Post by _delirium »

psanford wrote: post #34

Congrats Oxide team! More competition in this space is always a good thing. I'm curious about management. Can the rack operate completely standalone? I assume when you have multiple there will be some management abstraction above the rack layer? The closest direct equivalent that I can think of to this is AWS outposts. Are there any others that I'm forgetting?

The density they're getting here is significantly higher than AWS Outposts, which is interesting. The top-end (~$600k) AWS Outposts seem to max out at around 1k CPUs and 4.5 TB RAM in a rack (e.g. 12x m5.24xlarge = 12x 384 GB), while this rack can house 2k CPUs and 30 TB (!) RAM.

Re: Servers as they should be – shipping early 2022

#49

Post by kaliszad »

ptomato wrote: post #26
boulos wrote: post #21

First, congrats! But second, I'd love to understand the compute vs storage tradeoff chosen here. Looking at the (pretty!) picture [1], I was shocked to see "Wow, it's mostly storage?". Is that from going all flash? Heading to https://oxide.computer/product for more details, lists: - 2048 cores - 30 TB of memory - 1024 TB of flash (1 PiB) Given how much of the rack is storage, I'm not sure which Milan was chosen (and…

It looks like they're doing 2u half width nodes, so I'd strongly suspect each node is 1TB of ram, one epyc 7713p, and 10 3.2TB u.2/u.3 drives. eta: also suspect 30TB total just means they're leaving 64GB ram for the hypervisor OS on each node.

Leaving that RAM for ZFS L2 ARC perhaps? I don't think they would use Illumos as the hypervisor OS without also using OpenZFS with it. They also need some for management, the control UI, a DB for metrics and more.

Btw. if I count correctly, they have 20 SSD slots per node (if a node is full width) and 16 nodes. They would need 2 TB to reach 1 PB of "raw" capacity with the obvious redundancy overhead of ~ 20%.

It is also quite possible, they don't use ZFS at all and use e.g. Ceph or something like it but I don't think that is the case, because that wouldn't be cantrillian. :-) E.g. using Minio, they can provide something S3 like on top of a cluster of ZFS storage nodes too but they most likely get better latency with local ZFS and not a distributed filesystem. Financial institutions especially seem to be part of the target here and there latency can be king.

Re: Servers as they should be – shipping early 2022

#50

Post by zozbot234 »

wmf wrote: post #33
rbanffy wrote: post #30

Looks interesting. I wonder if the high integration makes it behave as a single-image machine with that many cores.

No, single system image is really expensive (see Superdome Flex) and most workloads can't justify that cost.

Could one reimplement SSI at the OS layer, similar to existing distributed OS's? Distributed shared memory is usually dismissed because of the overhead involved in typical scenarios, but this kind of hardware platform might make it feasible.
Post reply on HN