Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Servers as they should be – shipping early 2022

oxide.computer

Post reply on HN
Print view View original

21–30 of 264 posts

Re: Servers as they should be – shipping early 2022

#21

Post by boulos »

First, congrats!

But second, I'd love to understand the compute vs storage tradeoff chosen here. Looking at the (pretty!) picture [1], I was shocked to see "Wow, it's mostly storage?". Is that from going all flash?

Heading to https://oxide.computer/product for more details, lists:

- 2048 cores

- 30 TB of memory

- 1024 TB of flash (1 PiB)

Given how much of the rack is storage, I'm not sure which Milan was chosen (and so whether that's 2048 threads or 4196 [edit: real cores, 4196 threads]), but it seems like visually 4U is compute? [edit: nope] Is that a mistake on my part, because dual-socket Milan at 128 threads per socket is 256 threads per server, so you need at least 8 servers to hit 2048 "somethings", or do the storage nodes also have Milans [would make sense] and their compute is included [also fine!] -- and so similarly that's how you get a funky 30 TiB of memory?

[Top-level edit from below: the green stuff are the nodes, including the compute. The 4U near the middle is the fiber]

P.S.: the "NETWORK SPEED 100 GB/S" in all caps / CSS loses the presumably 100 Gbps (though the value in the HTML is 100 gb/s which is also unclear).

[1] https://oxide.computer/_next/image?url=%2Fimages%2Frenders%2...

Re: Servers as they should be – shipping early 2022

#22

Post by hlandau »

"Attests the software version is the version that is valid and shipped by the Oxide Computer Company"

So in other words these servers will implement restrictive code signing practices and will be vendor-controlled, not owner-controlled?

This is not my idea of "secure", and really in the wake of things like the Solarwinds or RSA hacks it shouldn't be anyone's idea of secure. Vendor-holds-the-keys is not an acceptable security model.

A comment below mentions open firmware, open firmware is useless without the right to deploy modified versions of it.

Happy to take clarification on this.

Re: Servers as they should be – shipping early 2022

#23

Post by 2trill2spill »

loudmax wrote: post #17

As I understand it, Oxide is going to have deep software integration into their hardware. So the expectation isn't that the servers in this rack will be running Windows or a generic Linux distribution. In case anyone from Oxide is here, is my understanding correct? And if so, will there be a way to run a smaller version of an Oxide system, say for testing or development, without purchasing an entire rack at a time? A…

My understanding is you will use a API to provision virtual machines on top of the Oxide hypervisor/software stack, which is bhyve running on Illumos. So you can still just run your favorite Linux distro or windows or a BSD if you want[1].

[1]: https://soundcloud.com/user-760920229/why-your-servers-suck-...

Re: Servers as they should be – shipping early 2022

#25

Post by jhickok »

loudmax wrote: post #17

As I understand it, Oxide is going to have deep software integration into their hardware. So the expectation isn't that the servers in this rack will be running Windows or a generic Linux distribution. In case anyone from Oxide is here, is my understanding correct? And if so, will there be a way to run a smaller version of an Oxide system, say for testing or development, without purchasing an entire rack at a time? A…

Agreed, I would love to hear more about the management plane. I'm glad it's API-driven, but I still have some questions about things like which hypervisor they are using.

If it's a custom software stack, might be nice to get a miniature dev-kit!

Re: Servers as they should be – shipping early 2022

#26

Post by ptomato »

boulos wrote: post #21

First, congrats! But second, I'd love to understand the compute vs storage tradeoff chosen here. Looking at the (pretty!) picture [1], I was shocked to see "Wow, it's mostly storage?". Is that from going all flash? Heading to https://oxide.computer/product for more details, lists: - 2048 cores - 30 TB of memory - 1024 TB of flash (1 PiB) Given how much of the rack is storage, I'm not sure which Milan was chosen (and…

It looks like they're doing 2u half width nodes, so I'd strongly suspect each node is 1TB of ram, one epyc 7713p, and 10 3.2TB u.2/u.3 drives.

eta: also suspect 30TB total just means they're leaving 64GB ram for the hypervisor OS on each node.

Re: Servers as they should be – shipping early 2022

#27

Post by richardwhiuk »

boulos wrote: post #21

First, congrats! But second, I'd love to understand the compute vs storage tradeoff chosen here. Looking at the (pretty!) picture [1], I was shocked to see "Wow, it's mostly storage?". Is that from going all flash? Heading to https://oxide.computer/product for more details, lists: - 2048 cores - 30 TB of memory - 1024 TB of flash (1 PiB) Given how much of the rack is storage, I'm not sure which Milan was chosen (and…

Suspect each node is both storage and compute.

Guessing they aren't counting threads (they say "cores"), so 64 cores per socket, 128 cores per server, 16 servers => 2048 cores.

Re: Servers as they should be – shipping early 2022

#28

Post by dcolkitt »

lumost wrote: post #5
zapita wrote: post #2

Who is the target customer for this?

This level of integration surely won't come cheap, from what I recall of Server purchasing a target price of ~200-500k per rack would be expected on a TCO of roughly 2x the rack price over 3 years. (Assuming you are buying from Quanta/SuperMicro or other commodity integrator) It's possible the prices are different now, but you would need customers looking to drop > 1 million dollars in CapEx for the management capabi…

That reminds me a lot of the Sun Microsystem mega servers from 20 years ago. Those were kind of the cure-all solutions to high scalability web services before Google et al. pioneered cloud-like services on commodity software.
Post reply on HN