Why Oxide Chose Illumos

#1

Post by kblissett » Wed, Sep 11, 2024, 9:22 PM UTC

Why Oxide Chose Illumos
rfd.shared.oxide.computer

Re: Why Oxide Chose Illumos

#2

Post by Rendello » Wed, Sep 11, 2024, 10:00 PM UTC

I like the RFDs. Oxide just did a podcast episode on the process:

https://oxide.computer/podcasts/oxide-and-friends/2065190

Re: Why Oxide Chose Illumos

#3

Post by rtpg » Thu, Sep 12, 2024, 6:27 AM UTC

> There is not a significant difference in functionality between the illumos and FreeBSD implementations, since pulling patches downstream has not been a significant burden. Conversely, the more advanced OS primitives in illumos have resulted in certain bugs being fixed only there, having been difficult to upstream to FreeBSD.

curious about what bugs are being thought of there. Sounds like a very interesting situation to be in

Re: Why Oxide Chose Illumos

#4

Post by taspeotis » Thu, Sep 12, 2024, 6:29 AM UTC

I kagi’d Illumos and apparently Bryan Cantrill was a maintainer.

Bryan Cantrill is CTO of Oxide [1].

I assume that has no bearing on the choice, otherwise it would be mentioned in the discussion.

[1] https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...

Re: Why Oxide Chose Illumos

#5

Post by bonzini » Thu, Sep 12, 2024, 6:52 AM UTC

> QEMU is often the subject of bugs affecting its reliability and security.

{{citation needed}}?

When I ran the numbers in 2019, there hadn't been guest exploitable vulnerabilities that affected devices normally used for IaaS for 3 years. Pretty much every cloud outside the big three (AWS, GCE, Azure) runs on QEMU.

Here's a talk I gave about it that includes that analysis:

slides - https://kvm-forum.qemu.org/2019/kvmforum19-bloat.pdf

video - https://youtu.be/5TY7m1AneRY?si=Sj0DFpRav7PAzQ0Y

Re: Why Oxide Chose Illumos

#6

Post by gyre007 » Thu, Sep 12, 2024, 6:57 AM UTC

taspeotis wrote: post #4

I kagi’d Illumos and apparently Bryan Cantrill was a maintainer. Bryan Cantrill is CTO of Oxide [1]. I assume that has no bearing on the choice, otherwise it would be mentioned in the discussion. [1] https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...

Yeah I came here to say that Bryan worked at Sun so why do they even need to write this post (yes, I appreciate the techinical reasons, just wanted to highlight the fact via a subtle dig :-))

Re: Why Oxide Chose Illumos

#7

Post by sausagefeet » Thu, Sep 12, 2024, 6:58 AM UTC

taspeotis wrote: post #4

I kagi’d Illumos and apparently Bryan Cantrill was a maintainer. Bryan Cantrill is CTO of Oxide [1]. I assume that has no bearing on the choice, otherwise it would be mentioned in the discussion. [1] https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...

Early Oxide founders came from Joyent which was an illumos shop and Cantrill is quite vocal about the history of Solaris, OpenSolaris, and illumos.

Re: Why Oxide Chose Illumos

#8

Post by sausagefeet » Thu, Sep 12, 2024, 7:00 AM UTC

gyre007 wrote: post #6

taspeotis wrote: post #4

I kagi’d Illumos and apparently Bryan Cantrill was a maintainer. Bryan Cantrill is CTO of Oxide [1]. I assume that has no bearing on the choice, otherwise it would be mentioned in the discussion. [1] https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...

Yeah I came here to say that Bryan worked at Sun so why do they even need to write this post (yes, I appreciate the techinical reasons, just wanted to highlight the fact via a subtle dig :-))

This isn't a blog post from an Oxide, it's a link to their internal RFD which they use to make decisions.

Re: Why Oxide Chose Illumos

#9

Post by sausagefeet » Thu, Sep 12, 2024, 7:06 AM UTC

While it's fair to say this does describe why Illumos was chosen, the actual RFD title is not presented and it is about Host OS + Virtualization software choice.

Even if you think it's a foregone conclusion given the history of bcantrill and other founders of Oxide, there absolutely is value in putting decision to paper and trying to provide a rational because then it can be challenged.

The company I co-founded does an RFD process as well and even if there is 99% chance that we're going to use the thing we've always used, if you're a serious person, the act of expressing it is useful and sometimes you even change your own mind thanks to the process.

Re: Why Oxide Chose Illumos

#10

Post by transpute » Thu, Sep 12, 2024, 7:16 AM UTC

> Xen: Large and complicated (by dom0) codebase, discarded for KVM by AMZN

  1. Xen Type-1 hypervisor is smaller than KVM/QEMU.
  2. Xen "dom0" = Linux/FreeBSD/OpenSolaris. KVM/bhyve also need host OS.
  3. AMZN KVM-subset: x86 cpu/mem virt, blk/net via Arm Nitro hardware.
  4. bhyve is Type-2.
  5. Xen has Type-2 (uXen).
  6. Xen dom0/host can be disaggregated (Hyperlaunch), unlike KVM.
  7. pKVM (Arm/Android) is smaller than KVM/Xen.

> The Service Management Facility (SMF) is responsible for the supervision of services under illumos.. a [Linux] robust infrastructure product would likely end up using few if any of the components provided by the systemd project, despite there now being something like a hundred of them. Instead, more traditional components would need to be revived, or thoroughly bespoke software would need to be developed, in order to avoid the technological and political issues with this increasingly dominant force in the Linux ecosystem.

Is this an argument for Illumos over Linux, or for translating SMF to Linux?