I guess that's one way of keeping Solaris alive :)
Why Oxide Chose Illumos
31–40 of 177 posts
Re: Why Oxide Chose Illumos
#32"• Emerging VMMs (OpenBSD’s vmm, etc): Haven’t been proven in production"
It's a small operation, but https://openbsd.amsterdam/ have absolutely proven that OpenBSD's hypervisor is production-capable in terms of stability - but there are indeed other problems that rule against it on scale.
For those who are unfamiliar with OpenBSD: the primary caveat is that its hypervisor can so far only provide guests with a single CPU core.
Re: Why Oxide Chose Illumos
#33> Nested virtualisation [...] challenging to emulate the underlying interfaces with flawless fidelity [...] dreadful performance
It is so sad that we've ended up with designs where this is the case. There is no intrinsic reason why nested virtualization should be hard to implement or should perform poorly. Path dependence strikes again.
Re: Why Oxide Chose Illumos
#34> Nested virtualisation [...] challenging to emulate the underlying interfaces with flawless fidelity [...] dreadful performance It is so sad that we've ended up with designs where this is the case. There is no intrinsic reason why nested virtualization should be hard to implement or should perform poorly. Path dependence strikes again.
It doesn't perform poorly in fact. It can be tuned at 90% of non-nested virtualization, and for workloads where it doesn't, that's more than anything else a testimony to how close virtualized performance is to bare-metal.
That said, it does add a lot of complexity.
Re: Why Oxide Chose Illumos
#35Instead of stating more or less irrelevant reasons, I'd prefer to read something like "I am (have been?) one of the core maintainers and know Illumos and Bhyve, so even if there would be 'objectively' better choices, our familiarity with the OS and hypervisor trump that". A "I like $A, always use $A and have experience using $A" is almost always a better argument than "$A is better than $B because $BLA", because that…
But Bryan also ported KVM to Illumos. And Joyand used KVM and they supported KVM there for years, I assume Bryan knows more about KVM then Bhyve as he seemed very hands on in the implementation (there is nice talk on youtube). So the idea that he isn't familiar with KVM isn't the case. So based on that KVM or Bhyve on Illumos, KVM would suggest itself. In the long term if $A is actually better then $B, then it makes…
For what it's worth, we at _Joyent_ were seriously investing in bhyve as our next generation of hypervisor for quite a while. We had been diverging from upstream KVM, and most especially upstream QEMU, for a long time, and bhyve was a better fit for us for a variety of reasons. We adopted a port that had begun at Pluribus, another company that was doing things with OpenSolaris and eventually illumos, and Bryan lead us through that period as well.
Re: Why Oxide Chose Illumos
#36These sound like reason you retconned so it sounds like you didn't choose Illumos because your founder used to work at Sun and Joyent before. :-)
Frankly I don't understand why they blogged that at all. It reeks of desperation, like they feel they need to defend their choice. They don't.
It also should not matter to their customers. They get exposed APIs and don't have to care about the implementation details.
Re: Why Oxide Chose Illumos
#37Earlier quoted context omitted.
The underlying hyperwiser on oxide isn't exposed to the consumers of the API. Just like on Amazon. I think arguably the bhyve over KVM was the more fundamental reason, and bhyve doesn't run on linux anyway.
Exactly, then why would they be dragged into systemd-or-not-systemd discussion? If you want to use Linux, use either Debian or the CentOS hyperscaler spin (the one that Meta uses) and call it a day. I am obviously biased as I am a KVM (and QEMU) developer myself, but I don't see any other plausible reason other than "we know the Illumos userspace best". Founder mode and all that. As to their choice of hypervisor, to…
FWIW, founder mode didn't exist five years ago when we were getting started! More seriously, though, this document (which I helped write) is an attempt specifically to avoid classic FUD tropes. It's not perfect, but it reflects certainly aspects of my lived experience in trying to get pieces of the Linux ecosystem to work in production settings.
While it's true that I'm a dyed in the wool illumos person, being in the core team and so on, I have Linux desktops, and the occasional Linux system in lab environments. I have been supporting customers with all sorts of environments that I don't get to choose for most of my career, including Linux and Windows systems. At Joyent most of our customers were running hardware virtualised Linux and Windows guests, so it's not like I haven't had a fair amount of exposure. I've even spent several days getting SCO OpenServer to run under our KVM, for a customer, because I apparently make bad life choices!
As for not discussing the social and political stuff in any depth, I felt at the time (and still do today) that so much ink had been split by all manner of folks talking about LKML or systemd project behaviour over the last decade that it was probably a distraction to do anything other than mention it in passing. As I believe I said in the podcast we did about this RFD recently: I'm not sure if this decision would be right for anybody else or not, but I believe it was and is right for us. I'm not trying to sell you, or anybody else, on making the same calls. This is just how we made our decision.
Re: Why Oxide Chose Illumos
#38I kagi’d Illumos and apparently Bryan Cantrill was a maintainer. Bryan Cantrill is CTO of Oxide [1]. I assume that has no bearing on the choice, otherwise it would be mentioned in the discussion. [1] https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...
[flagged]
Re: Why Oxide Chose Illumos
#39I’d love to use Illumos, but a lack of arm64 support is a non-starter
Re: Why Oxide Chose Illumos
#40"• Emerging VMMs (OpenBSD’s vmm, etc): Haven’t been proven in production" It's a small operation, but https://openbsd.amsterdam/ have absolutely proven that OpenBSD's hypervisor is production-capable in terms of stability - but there are indeed other problems that rule against it on scale. For those who are unfamiliar with OpenBSD: the primary caveat is that its hypervisor can so far only provide guests with a single…
Yes, to be clear this is not meant to be a criticism of software quality at OpenBSD! Though I don't necessarily always agree with the leadership style I have big respect for their engineering efforts and obviously as another relatively niche UNIX I feel a certain kinship! That part of the document was also written some years ago, much closer to 2018 when that service got started than now, so it's conceivable that we wouldn't have said the same thing today.
I will say, though, that single VCPU guests would not have met our immediate needs in the Oxide product!