Live data from Hacker News

Why Oxide Chose Illumos

rfd.shared.oxide.computer

21–30 of 177 posts

Re: Why Oxide Chose Illumos

#21

Instead of stating more or less irrelevant reasons, I'd prefer to read something like "I am (have been?) one of the core maintainers and know Illumos and Bhyve, so even if there would be 'objectively' better choices, our familiarity with the OS and hypervisor trump that". A "I like $A, always use $A and have experience using $A" is almost always a better argument than "$A is better than $B because $BLA", because that…

But Bryan also ported KVM to Illumos. And Joyand used KVM and they supported KVM there for years, I assume Bryan knows more about KVM then Bhyve as he seemed very hands on in the implementation (there is nice talk on youtube). So the idea that he isn't familiar with KVM isn't the case. So based on that KVM or Bhyve on Illumos, KVM would suggest itself.

In the long term if $A is actually better then $B, then it makes sense to start with $A even if you don't know $A. Because if you are trying to building a company that is hopefully making billions in revenue in the future, then long term matters a great deal.

Now the question is can you objectively figure out if $A or $B is better. And how much time does it take to figure out. Familiarity of the team is one consideration but not the most important one.

Trying to be objective about this, instead of just saying 'I know $A' seems quite like a smart thing to do. And writing it down also seems smart.

In a few years you can look back and actually say, was our analysis correct, if no what did we misjudge. And then you can learn from that.

If you just go with familiarity you are basically saying 'our failure was predetermined so we did nothing wrong', when you clearly did go wrong.

Re: Why Oxide Chose Illumos

#22

Earlier quoted context omitted.

But everyone at Oxide already knows that back story. At least if you list some other reasons list you can have a discussion about technical merits if you want to.

But that doesn't make sense if you have specialists for $A that also like to work with $A. Why should I as a customer trust Illumos/Bhye developers that are using Linux/KVM instead of "real" Linux/KVM developers? The only thing that such a decision would tell me is to not even think about using Illumos or Bhyve. The difference between "Buy our Illumos/Bhye solution! Why? I have been an Illumos/Bhyve Maintainer!" and…

Those are not the only options. You can have KVM on Illumos, or Bhye on FreeBSD.

And finding people to heir that know Linux/KVM wouldn't be a problem for them.

This evaluation was done years ago and they added like 50 people since then.

Saying 'We have a great KVM Team but our CEO was once an Illumos developer' is perfectly reasonable.

And as I point out in my other comment, the former Joyant people like know more about KVM then anything else anyway. So it would be:

"Buy our KVM Solution, we have KVM experts"

But they evaluated that Bhyve was better then KVM despite that.

Re: Why Oxide Chose Illumos

#23
post #12

Earlier quoted context omitted.

Talking about "technological and political issues" without mentioning any, or without mentioning which components would need to be revived, sounds a lot like FUD unfortunately. Mixing and matching traditional and systemd components is super common, for example Fedora and RHEL use chrony instead of timesyncd, and NetworkManager instead of networkd.

I read it as "we can sit in this more quiet room where people don't rave about systemd all day long".

But do they? Oxide targets the enterprise, and people there don't care that much about how the underlying OS works. It's been ten years since a RHEL release started using systemd and there has been no exodus to either Windows or Illumos.

I don't mean FUD in a disparaging sense, more like literal fear of the unknown causing people to be excessively cautious. I wouldn't have any problem with Oxide saying "we went for what we know best", there's no need to fake that so much more research went into a decision.

Re: Why Oxide Chose Illumos

#24

I kagi’d Illumos and apparently Bryan Cantrill was a maintainer. Bryan Cantrill is CTO of Oxide [1]. I assume that has no bearing on the choice, otherwise it would be mentioned in the discussion. [1] https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...

Bryan Cantrill also ported KVM to Illumos. At Joyent they had plenty of experience with KVM. See:

https://www.youtube.com/watch?v=cwAfJywzk8o

As far as I know, Bryan didn't personally work on the porting of bhyve (this might be wrong).

So if anything, that would point to KVM as the 'familiar' thing given how many former Joyant people were there.

Re: Why Oxide Chose Illumos

#25

Earlier quoted context omitted.

Early Oxide founders came from Joyent which was an illumos shop and Cantrill is quite vocal about the history of Solaris, OpenSolaris, and illumos.

> Joyent which was an illumos shop And before that, they used to run FreeBSD. Mentioned for example in this comment by Bryan Cantrill a decade ago: https://news.ycombinator.com/item?id=6254092 > […] Speaking only for us (I work for Joyent), we have deployed hundreds of thousands of zones into production over the years -- and Joyent was running with FreeBSD jails before that […] And I’ve seen some other primary source…

Joyent was using Solaris before Bryan worked there. Listen to the this podcast with Bryan and his co-founder about their origin story:

https://www.youtube.com/watch?v=eVkIKm9pkPY

This is about as good as you are gone get on the topic of Joyant history.

Re: Why Oxide Chose Illumos

#26

Earlier quoted context omitted.

But that doesn't make sense if you have specialists for $A that also like to work with $A. Why should I as a customer trust Illumos/Bhye developers that are using Linux/KVM instead of "real" Linux/KVM developers? The only thing that such a decision would tell me is to not even think about using Illumos or Bhyve. The difference between "Buy our Illumos/Bhye solution! Why? I have been an Illumos/Bhyve Maintainer!" and…

Those are not the only options. You can have KVM on Illumos, or Bhye on FreeBSD. And finding people to heir that know Linux/KVM wouldn't be a problem for them. This evaluation was done years ago and they added like 50 people since then. Saying 'We have a great KVM Team but our CEO was once an Illumos developer' is perfectly reasonable. And as I point out in my other comment, the former Joyant people like know more ab…

> "Buy our KVM Solution, we have KVM experts"

Of course, but that is less of unique selling point.

> But they evaluated that Bhyve was better then KVM despite that.

If you are selling Bhyve you better say that whether it's true or not. So why should I, as a reader or employee or customer, trust them?

Re: Why Oxide Chose Illumos

#27
post #23

Earlier quoted context omitted.

I read it as "we can sit in this more quiet room where people don't rave about systemd all day long".

But do they? Oxide targets the enterprise, and people there don't care that much about how the underlying OS works. It's been ten years since a RHEL release started using systemd and there has been no exodus to either Windows or Illumos. I don't mean FUD in a disparaging sense, more like literal fear of the unknown causing people to be excessively cautious. I wouldn't have any problem with Oxide saying "we went for w…

The underlying hyperwiser on oxide isn't exposed to the consumers of the API. Just like on Amazon.

I think arguably the bhyve over KVM was the more fundamental reason, and bhyve doesn't run on linux anyway.

Re: Why Oxide Chose Illumos

#28

Earlier quoted context omitted.

> Joyent which was an illumos shop And before that, they used to run FreeBSD. Mentioned for example in this comment by Bryan Cantrill a decade ago: https://news.ycombinator.com/item?id=6254092 > […] Speaking only for us (I work for Joyent), we have deployed hundreds of thousands of zones into production over the years -- and Joyent was running with FreeBSD jails before that […] And I’ve seen some other primary source…

Joyent was using Solaris before Bryan worked there. Listen to the this podcast with Bryan and his co-founder about their origin story: https://www.youtube.com/watch?v=eVkIKm9pkPY This is about as good as you are gone get on the topic of Joyant history.

Thank you, I will watch that right away :)

Re: Why Oxide Chose Illumos

#29
post #23

Earlier quoted context omitted.

But do they? Oxide targets the enterprise, and people there don't care that much about how the underlying OS works. It's been ten years since a RHEL release started using systemd and there has been no exodus to either Windows or Illumos. I don't mean FUD in a disparaging sense, more like literal fear of the unknown causing people to be excessively cautious. I wouldn't have any problem with Oxide saying "we went for w…

The underlying hyperwiser on oxide isn't exposed to the consumers of the API. Just like on Amazon. I think arguably the bhyve over KVM was the more fundamental reason, and bhyve doesn't run on linux anyway.

Exactly, then why would they be dragged into systemd-or-not-systemd discussion? If you want to use Linux, use either Debian or the CentOS hyperscaler spin (the one that Meta uses) and call it a day.

I am obviously biased as I am a KVM (and QEMU) developer myself, but I don't see any other plausible reason other than "we know the Illumos userspace best". Founder mode and all that.

As to their choice of hypervisor, to be honest KVM on Illumos was probably not a great idea to begin with, therefore they used bhyve.

Re: Why Oxide Chose Illumos

#30

I kagi’d Illumos and apparently Bryan Cantrill was a maintainer. Bryan Cantrill is CTO of Oxide [1]. I assume that has no bearing on the choice, otherwise it would be mentioned in the discussion. [1] https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...

Bryan Cantrill also ported KVM to Illumos. At Joyent they had plenty of experience with KVM. See: https://www.youtube.com/watch?v=cwAfJywzk8o As far as I know, Bryan didn't personally work on the porting of bhyve (this might be wrong). So if anything, that would point to KVM as the 'familiar' thing given how many former Joyant people were there.

KVM got more and more integrated with the rest of Linux as more virtualization features became general system features (e.g. posted interrupts). Also Google and Amazon are working more upstream and the pace of development increased a lot.

Keeping a KVM port up to date is a huge effort compared to bhyve, and they probably had learnt that in the years between the porting of KVM and the founding of Oxide.

Post reply on HN