Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Helios: A distribution of Illumos powering the Oxide Rack

github.com

Post reply on HN
Print view View original

31–40 of 268 posts

Re: Helios: A distribution of Illumos powering the Oxide Rack

#31

Post by refulgentis »

qmarchi wrote: post #27
greggyb wrote: post #23

Earlier quoted context omitted.

If you're running in one of the big 3 cloud providers, the bottom-level hypervisors are not-linux. This is equivalent. Are you anti-AWS or anti-Azure for the same reason? This is the substrate upon which you will run any virtualized infrastructure.

Small note, that's not true for Google Cloud, which runs on top of Linux, though modified. Disclaimer: Former Googler, Cloud Support

Another Xoogler here: any idea what they mean by it's not Linux at the bottom for other providers? Like, surely it's _some_ common OS? Either my binaries wouldn't run or AWS is reimplementing Linux so they can, which seems odd.

Or are they just saying that the VM my binary runs on might be some predictable Linux version, but the underlying thing launching the VM could be anything?

Re: Helios: A distribution of Illumos powering the Oxide Rack

#32

Post by [deleted] »

refulgentis wrote: post #29
milon wrote: post #24

I'm glad this is out, i'm going to deploy this locally and learn as much about it as possible. Oxide is pretty much the company I dream to work at, both for the tech stack, plus the people working there. Thank you Oxide team!

Can you get me excited? I spent 20 seconds browsing the homepage and walked away with "so the idea is vertical integration for on-premise server purchases? On custom OS? Why? Why would people pay a premium?" But immediately got myself to "what does a server OS do anyway, doesn't it just launch VMs? You don't need Linux, just the ability to launch Linux VMs" Tell me more? :)

[deleted]

Re: Helios: A distribution of Illumos powering the Oxide Rack

#33

Post by throwup238 »

refulgentis wrote: post #29
milon wrote: post #24

I'm glad this is out, i'm going to deploy this locally and learn as much about it as possible. Oxide is pretty much the company I dream to work at, both for the tech stack, plus the people working there. Thank you Oxide team!

Can you get me excited? I spent 20 seconds browsing the homepage and walked away with "so the idea is vertical integration for on-premise server purchases? On custom OS? Why? Why would people pay a premium?" But immediately got myself to "what does a server OS do anyway, doesn't it just launch VMs? You don't need Linux, just the ability to launch Linux VMs" Tell me more? :)

The best elevator pitch I've heard is "AWS APIs for on-prem datacenters". They make turn-key managed racks that behave just like a commercial cloud would with all the APIs for VM, storage, and network provisioning and integration you'd expect from AWS, except made to deploy in your company's datacenter under your control.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#34

Post by bcantrill »

StillBored wrote: post #26
sneak wrote: post #2

I know they’re ex-Sun, but is there any real technical benefit for choosing not-Linux (for their business value prop)? I know of the technical benefits of illumos over linux, but does that actually matter to the customers who are buying these? Aren’t they opening a whole can of worms for ideology/tradition that won’t sell any more computers? As someone who runs Linux container workloads, the fact that this is fundame…

Linux is a nightmare in the embedded/appliance space because one ends up just having platform engineers who spend their day fixing problems with the latest kernels, drivers, core libraries, etc, that the actual application depends on. Or one goes the route of 99% of the IoT/etc vendors, and never update the base OS and pray that there aren't any active exploits targeting it. This is why a lot of medium-sized companie…

This is a really, really good point -- and is a result of the model of Linux being only a kernel (and not system libraries, commands, etc.). It means that any real use of Linux is not merely signing up for kernel maintenance (which itself can be arduous) but also must make decisions around every other aspect of the system (each with its own communities, release management, etc.). This act is the act of creating a distribution -- and it's a huge burden to take on. Both illumos and the BSD derivatives make this significantly easier by simply including much more of the system within their scope: they are not merely kernels, but also system libraries and commands.

This weighed heavily in our own calculus, so I'm glad you brought it up!

Re: Helios: A distribution of Illumos powering the Oxide Rack

#35

Post by qmarchi »

refulgentis wrote: post #31
qmarchi wrote: post #27

Earlier quoted context omitted.

Small note, that's not true for Google Cloud, which runs on top of Linux, though modified. Disclaimer: Former Googler, Cloud Support

Another Xoogler here: any idea what they mean by it's not Linux at the bottom for other providers? Like, surely it's _some_ common OS? Either my binaries wouldn't run or AWS is reimplementing Linux so they can, which seems odd. Or are they just saying that the VM my binary runs on might be some predictable Linux version, but the underlying thing launching the VM could be anything?

Correct, that the Hypervisor isn't running Linux.

I think the only provider where that would make sense would be Microsoft, where they have their own OS.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#36

Post by amluto »

wmf wrote: post #22
steveklabnik wrote: post #14

Earlier quoted context omitted.

> does that actually matter to the customers who are buying these? It's not like we specifically say "oh btw there's illumos inside and that's why you should buy the rack." It's not a customer-facing detail of the product. I'm sure most will never even know that this is the case. What customers do care about is that the rack is efficient, reliable, suits their needs, etc. Choosing illumos instead of Linux here is a c…

The Linux vs. Illumos decision seems to be downstream of a more fundamental decision to make VMs the narrow waist of the Oxide system. That's what I'm curious about.

Especially since Oxide has a big fancy firmware stack. I would expect this stack to be able to do an excellent job of securely allocating bare-metal (i.e. VMX root on x86 or EL2 if Oxide ever goes ARM) resources.

This would allow workloads on Oxide to run their own VMs, to safely use PCIe devices without dealing with interrupt redirection, etc.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#37

Post by kortilla »

throwup238 wrote: post #33
refulgentis wrote: post #29

Earlier quoted context omitted.

Can you get me excited? I spent 20 seconds browsing the homepage and walked away with "so the idea is vertical integration for on-premise server purchases? On custom OS? Why? Why would people pay a premium?" But immediately got myself to "what does a server OS do anyway, doesn't it just launch VMs? You don't need Linux, just the ability to launch Linux VMs" Tell me more? :)

The best elevator pitch I've heard is "AWS APIs for on-prem datacenters". They make turn-key managed racks that behave just like a commercial cloud would with all the APIs for VM, storage, and network provisioning and integration you'd expect from AWS, except made to deploy in your company's datacenter under your control.

That’s the elevator pitch for open stack

Re: Helios: A distribution of Illumos powering the Oxide Rack

#38

Post by milon »

refulgentis wrote: post #29
milon wrote: post #24

I'm glad this is out, i'm going to deploy this locally and learn as much about it as possible. Oxide is pretty much the company I dream to work at, both for the tech stack, plus the people working there. Thank you Oxide team!

Can you get me excited? I spent 20 seconds browsing the homepage and walked away with "so the idea is vertical integration for on-premise server purchases? On custom OS? Why? Why would people pay a premium?" But immediately got myself to "what does a server OS do anyway, doesn't it just launch VMs? You don't need Linux, just the ability to launch Linux VMs" Tell me more? :)

Having a solid on-prem rack product to me is a great thing. I like IaaS services a lot, don't get me wrong, and I think they're the right pick for a bunch of cases, but on-prem servers also have their "place in the sun", so to speak :) I could present any number of justifications that I don't think I'm qualified enough to defend, but the gist is that at the bare minimum, I'm glad the option exists.

As to why I'm personally excited: I enjoy the amount of control having such an on-prem rack would afford me, and there surely could be a great amount of cost-savings and energy-savings in many scenarios. Sometimes, you just need a rack to deploy services for your local business. I like the prospect of decentralizing infrastructure, applying all the things we've learned with IaaSes.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#39

Post by p_l »

refulgentis wrote: post #31
qmarchi wrote: post #27

Earlier quoted context omitted.

Small note, that's not true for Google Cloud, which runs on top of Linux, though modified. Disclaimer: Former Googler, Cloud Support

Another Xoogler here: any idea what they mean by it's not Linux at the bottom for other providers? Like, surely it's _some_ common OS? Either my binaries wouldn't run or AWS is reimplementing Linux so they can, which seems odd. Or are they just saying that the VM my binary runs on might be some predictable Linux version, but the underlying thing launching the VM could be anything?

Old AWS used to be Xen, Nitro afaik uses customised VMM and I don't recall if it's not a custom OS or hosted on top of something.

Azure is Hyper-V underneath IIRC, a custom variant at least (remember Windows Server Nano? IIRC it was the closest you could get to running it), with sometimes weird things like network cards running Linux and integrating with Windows' built-in SDN facility.

Rest of the bigger ones is mainly Linux with occasional Xen and such, but sometimes you can encounter non-trivial VMware deployments.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#40

Post by wmf »

amluto wrote: post #36
wmf wrote: post #22

Earlier quoted context omitted.

The Linux vs. Illumos decision seems to be downstream of a more fundamental decision to make VMs the narrow waist of the Oxide system. That's what I'm curious about.

Especially since Oxide has a big fancy firmware stack. I would expect this stack to be able to do an excellent job of securely allocating bare-metal (i.e. VMX root on x86 or EL2 if Oxide ever goes ARM) resources. This would allow workloads on Oxide to run their own VMs, to safely use PCIe devices without dealing with interrupt redirection, etc.

I'm not affiliated with Oxide but I don't think you can put Crucible and VPC/OPTE in firmware. Without a DPU those components have to run in the hypervisor.
Post reply on HN