Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Oxide Chose Illumos

rfd.shared.oxide.computer

Post reply on HN
Print view View original

91–100 of 177 posts

Re: Why Oxide Chose Illumos

#91

Post by 9front »

bonzini wrote: post #50
_rs wrote: post #47

Earlier quoted context omitted.

I thought AWS uses KVM, which is the same VM that QEMU would use? Or am I mistaken?

AWS uses KVM in the kernel but they have a different, non-open source userspace stack for EC2; plus Firecracker which is open source but is only used for Lambda, and runs on EC2 bare metal instances. Google also uses KVM with a variety of userspace stacks: a proprietary one (tied to a lot of internal Google infrastructure but overall a lot more similar to QEMU than Amazon's) for GCE, gVisor for AppEngine or whatever…

EC2 instances are using the Xen hypervisor. At least that's what reported by hostnamectl.

Re: Why Oxide Chose Illumos

#92

Post by pclmulqdq »

sophacles wrote: post #61
alberth wrote: post #57

Isn’t it simply Oxide founders are old Sun engineers, and Illumos is the open source spinoff of their old work.

According to the founders and early engineers on their podcast - no, they tried to fairly evaluate all the oses and were willing to go with other options. Practically speaking, its hard to do it completely objectively and the in-house expertise probably colored the decision.

Tried to, sure, but when you evaluate other products strictly against the criteria under which you built your own version, you know what the conclusion will be. Never mind that you are carrying your blind spots with you. I would say that there was an attempt to evaluate other products, but not so much an attempt to be objective in that evaluation.

In general, being on your own private tech island is a tough thing to do, but many engineers would rather do that than swallow their pride.

Re: Why Oxide Chose Illumos

#93

Post by geerlingguy »

jclulow wrote: post #42
leoh wrote: post #39

I’d love to use Illumos, but a lack of arm64 support is a non-starter

Folks are working on it! I believe it boots on some small systems and under QEMU, but it's still relatively early days. I'm excited for the port to eventually make it into the gate, though!

In before someone asks about riscv64 ;)

Re: Why Oxide Chose Illumos

#94

Post by mechanicker »

Wonder if this is more due to Bhyve being developed on FreeBSD and Illumos derives from a common ancestor BSD?

I know NetApp (stack based on FreeBSD) contributed significantly to Bhyve when they were exploring options to virtualize Data ONTAP (C mode)

https://forums.freebsd.org/threads/bhyve-the-freebsd-hypervi...

Re: Why Oxide Chose Illumos

#96

Post by ameliaquining »

otabdeveloper4 wrote: post #74
bonzini wrote: post #51

Earlier quoted context omitted.

That would be a damn good record though, isn't it? (I am fairly sure that more were found since, but the point is that these are pretty rare). Firecracker, which is written in Rust, had one in 2019: https://www.cve.org/CVERecord?id=CVE-2019-18960 Also QEMU's fuzzing is very sophisticated. Most recent vulnerabilities were found that way rather than by security researchers, which I don't think it's the case for "compet…

[flagged]

I do think that only having one CVE in six years is a pretty decent record, especially since that vulnerability probably didn't grant arbitrary code execution in practice.

Rust is an important part of how Firecracker pulls this off, but it's not the only part. Another important part is that it's a much smaller codebase than QEMU, so there are fewer places for bugs to hide. (This, in turn, is possible in part because Firecracker deliberately doesn't implement any features that aren't necessary for its core use case of server-side workload isolation, whereas QEMU aims to be usable for anything that you might want to use a VM for.)

Re: Why Oxide Chose Illumos

#97

Post by wmf »

9front wrote: post #91
bonzini wrote: post #50

Earlier quoted context omitted.

AWS uses KVM in the kernel but they have a different, non-open source userspace stack for EC2; plus Firecracker which is open source but is only used for Lambda, and runs on EC2 bare metal instances. Google also uses KVM with a variety of userspace stacks: a proprietary one (tied to a lot of internal Google infrastructure but overall a lot more similar to QEMU than Amazon's) for GCE, gVisor for AppEngine or whatever…

EC2 instances are using the Xen hypervisor. At least that's what reported by hostnamectl.

EC2 migrated off Xen around ten years ago. Only really old instances should be using Xen or Xen emulation.

Re: Why Oxide Chose Illumos

#98

Post by ComputerGuru »

jclulow wrote: post #35
panick21_ wrote: post #21

Earlier quoted context omitted.

But Bryan also ported KVM to Illumos. And Joyand used KVM and they supported KVM there for years, I assume Bryan knows more about KVM then Bhyve as he seemed very hands on in the implementation (there is nice talk on youtube). So the idea that he isn't familiar with KVM isn't the case. So based on that KVM or Bhyve on Illumos, KVM would suggest itself. In the long term if $A is actually better then $B, then it makes…

For what it's worth, we at _Joyent_ were seriously investing in bhyve as our next generation of hypervisor for quite a while. We had been diverging from upstream KVM, and most especially upstream QEMU, for a long time, and bhyve was a better fit for us for a variety of reasons. We adopted a port that had begun at Pluribus, another company that was doing things with OpenSolaris and eventually illumos, and Bryan lead u…

Are you/will you be upstreaming fixes and/or improvements to Bhyve?

Re: Why Oxide Chose Illumos

#99

Post by wmf »

throw0101b wrote: post #46

Somewhat related, they discussed why they chose to use ZFS for their storage backend as opposed to (say) Ceph in a podcast episode: * https://www.youtube.com/watch?v=UvEKSqBBcZw Certainly they already had experience with ZFS (as it is built into Illumos/Solaris), but as it was told to them by someone they trusted who ran a lot of Ceph: " Ceph is operated, not shipped [like ZFS] ". There's more care-and-feeding requir…

You mean they use Crucible instead of Ceph?

Re: Why Oxide Chose Illumos

#100

Post by ComputerGuru »

anonfordays wrote: post #88
throw0101b wrote: post #46

Somewhat related, they discussed why they chose to use ZFS for their storage backend as opposed to (say) Ceph in a podcast episode: * https://www.youtube.com/watch?v=UvEKSqBBcZw Certainly they already had experience with ZFS (as it is built into Illumos/Solaris), but as it was told to them by someone they trusted who ran a lot of Ceph: " Ceph is operated, not shipped [like ZFS] ". There's more care-and-feeding requir…

ZFS and Ceph is apples to oranges. ZFS is scoped to a single host, Ceph can span data centers.

It’s very possible to run a light/small layer on top of ZFS (either userspace daemon or via FUSE) to get you most of the way to scaling ZFS-backed object storage within or across data centers depending on what specific availability metrics you need.
Post reply on HN