Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Oxide Chose Illumos

rfd.shared.oxide.computer

Post reply on HN
Print view View original

41–50 of 177 posts

Re: Why Oxide Chose Illumos

#41

Post by jclulow »

fefe23 wrote: post #36

These sound like reason you retconned so it sounds like you didn't choose Illumos because your founder used to work at Sun and Joyent before. :-) Frankly I don't understand why they blogged that at all. It reeks of desperation, like they feel they need to defend their choice. They don't. It also should not matter to their customers. They get exposed APIs and don't have to care about the implementation details.

It's not a blog post, it's an RFD. We have a strong focus on writing as part of thinking and making decisions, and when we can, we like to publish our decision making documents in the spirit of open source. This is not a defence of our position so much as a record of the process through which we arrived at it. This is true of our other RFDs as well, which you can see on the site there.

> It also should not matter to their customers. They get exposed APIs and don't have to care about the implementation details.

Yes, the whole product is definitely designed that way intentionally. Customers get abstracted control of compute and storage resources through cloud style APIs. From their perspective it's a cloud appliance. It's only from our perspective as the people building it that it's a UNIX system.

Re: Why Oxide Chose Illumos

#42

Post by jclulow »

leoh wrote: post #39

I’d love to use Illumos, but a lack of arm64 support is a non-starter

Folks are working on it! I believe it boots on some small systems and under QEMU, but it's still relatively early days. I'm excited for the port to eventually make it into the gate, though!

Re: Why Oxide Chose Illumos

#43

Post by bonzini »

jclulow wrote: post #37
bonzini wrote: post #29

Earlier quoted context omitted.

Exactly, then why would they be dragged into systemd-or-not-systemd discussion? If you want to use Linux, use either Debian or the CentOS hyperscaler spin (the one that Meta uses) and call it a day. I am obviously biased as I am a KVM (and QEMU) developer myself, but I don't see any other plausible reason other than "we know the Illumos userspace best". Founder mode and all that. As to their choice of hypervisor, to…

FWIW, founder mode didn't exist five years ago when we were getting started! More seriously, though, this document (which I helped write) is an attempt specifically to avoid classic FUD tropes. It's not perfect, but it reflects certainly aspects of my lived experience in trying to get pieces of the Linux ecosystem to work in production settings. While it's true that I'm a dyed in the wool illumos person, being in the…

Founder mode existed, it just didn't have a catchy name. And I absolutely believe that it was the right choice for your team, exactly for "founder mode" reasons.

In other words, I don't think that the social or technological reasons in the document were that strong, and that's fine. Rather, my external armchair impression is simply that OS and hypervisor were not something where you were willing to spend precious "risk points", and that's the right thing to do given that you had a lot more places that were an absolute jump in the dark.

Re: Why Oxide Chose Illumos

#45

Post by elijahwright »

panick21_ wrote: post #24
taspeotis wrote: post #4

I kagi’d Illumos and apparently Bryan Cantrill was a maintainer. Bryan Cantrill is CTO of Oxide [1]. I assume that has no bearing on the choice, otherwise it would be mentioned in the discussion. [1] https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...

Bryan Cantrill also ported KVM to Illumos. At Joyent they had plenty of experience with KVM. See: https://www.youtube.com/watch?v=cwAfJywzk8o As far as I know, Bryan didn't personally work on the porting of bhyve (this might be wrong). So if anything, that would point to KVM as the 'familiar' thing given how many former Joyant people were there.

Where is Max Bruning these days?

Re: Why Oxide Chose Illumos

#46

Post by throw0101b »

Somewhat related, they discussed why they chose to use ZFS for their storage backend as opposed to (say) Ceph in a podcast episode:

* https://www.youtube.com/watch?v=UvEKSqBBcZw

Certainly they already had experience with ZFS (as it is built into Illumos/Solaris), but as it was told to them by someone they trusted who ran a lot of Ceph: "Ceph is operated, not shipped [like ZFS]".

There's more care-and-feeding required for it, and they probably don't want that as they want to treat product in a more appliance/toaster-like fashion.

Re: Why Oxide Chose Illumos

#47

Post by _rs »

bonzini wrote: post #5

> QEMU is often the subject of bugs affecting its reliability and security. {{citation needed}}? When I ran the numbers in 2019, there hadn't been guest exploitable vulnerabilities that affected devices normally used for IaaS for 3 years. Pretty much every cloud outside the big three (AWS, GCE, Azure) runs on QEMU. Here's a talk I gave about it that includes that analysis: slides - https://kvm-forum.qemu.org/2019/kvm…

I thought AWS uses KVM, which is the same VM that QEMU would use? Or am I mistaken?

Re: Why Oxide Chose Illumos

#48

Post by TimTheTinker »

bonzini wrote: post #5

> QEMU is often the subject of bugs affecting its reliability and security. {{citation needed}}? When I ran the numbers in 2019, there hadn't been guest exploitable vulnerabilities that affected devices normally used for IaaS for 3 years. Pretty much every cloud outside the big three (AWS, GCE, Azure) runs on QEMU. Here's a talk I gave about it that includes that analysis: slides - https://kvm-forum.qemu.org/2019/kvm…

> When I ran the numbers in 2019, there hadn't been guest exploitable vulnerabilities that affected devices normally used for IaaS for 3 years.

So there existed known guest-exploitable vulnerabilities as recently as 8 years ago. Maybe that, combined with the fact that QEMU is not written in Rust, is what is causing Oxide to decide against QEMU.

I think it's fair to say that any sufficiently large codebase originally written in C or C++ has memory safety bugs. Yes, the Oxide RFD author may be phrasing this using weasel words; and memory safety bugs may not be exploitable at a given point in a codebase's history. But I don't think that makes Oxide's decision invalid.

Re: Why Oxide Chose Illumos

#49

Post by jonstewart »

Illumos makes sense as a host OS—it’s capable, they know it, they can make sure it works well on their hardware, and virtualization means users don’t need that much familiarity with it.

If I were Oxide, though, I’d be sprinting to seamless VMWare support. Broadcom has turned into a modern-day Oracle (but dumber??) and many customers will migrate in the next two years. Even if those legacy VMs aren’t “hyperscale”, there’s going to be lots of budget devoted to moving off VMWare.

Re: Why Oxide Chose Illumos

#50

Post by bonzini »

_rs wrote: post #47
bonzini wrote: post #5

> QEMU is often the subject of bugs affecting its reliability and security. {{citation needed}}? When I ran the numbers in 2019, there hadn't been guest exploitable vulnerabilities that affected devices normally used for IaaS for 3 years. Pretty much every cloud outside the big three (AWS, GCE, Azure) runs on QEMU. Here's a talk I gave about it that includes that analysis: slides - https://kvm-forum.qemu.org/2019/kvm…

I thought AWS uses KVM, which is the same VM that QEMU would use? Or am I mistaken?

AWS uses KVM in the kernel but they have a different, non-open source userspace stack for EC2; plus Firecracker which is open source but is only used for Lambda, and runs on EC2 bare metal instances.

Google also uses KVM with a variety of userspace stacks: a proprietary one (tied to a lot of internal Google infrastructure but overall a lot more similar to QEMU than Amazon's) for GCE, gVisor for AppEngine or whatever it is called these days, crosvm for ChromeOS, and QEMU for Android Emulator.

Post reply on HN