Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Oxide Chose Illumos

rfd.shared.oxide.computer

Post reply on HN
Print view View original

121–130 of 177 posts

Re: Why Oxide Chose Illumos

#121

Post by simcop2387 »

9front wrote: post #112
wmf wrote: post #97

Earlier quoted context omitted.

EC2 migrated off Xen around ten years ago. Only really old instances should be using Xen or Xen emulation.

I'm puzzled by your comment. On an EC2 instance of AL2023 deployed on us-east-1 region this is the output of hostnamectl: [ec2-user][~]$ hostnamectl Static hostname: ip-x-x-x-x.ec2.internal Icon name: computer-vm Chassis: vm Machine ID: ec2d54f27fc534ea74980638ccc33d96 Boot ID: 6caf18b7ed3647819c1985c11f128142 Virtualization: xen Operating System: Amazon Linux 2023.5.20240903 CPE OS Name: cpe:2.3:o:amazon:amazon_linu…

I'm not quite sure the status of it at least, but reported back in 2017 that they are moving off Xen

https://www.theregister.com/2017/11/07/aws_writes_new_kvm_ba...

It could be that it's not all over and tied to specific machine types still, or there's something they've done to make it report to the guest still that it's xen based for some compatibility reasons.

Re: Why Oxide Chose Illumos

#122

Post by pmooney »

ComputerGuru wrote: post #98
jclulow wrote: post #35

Earlier quoted context omitted.

For what it's worth, we at _Joyent_ were seriously investing in bhyve as our next generation of hypervisor for quite a while. We had been diverging from upstream KVM, and most especially upstream QEMU, for a long time, and bhyve was a better fit for us for a variety of reasons. We adopted a port that had begun at Pluribus, another company that was doing things with OpenSolaris and eventually illumos, and Bryan lead u…

Are you/will you be upstreaming fixes and/or improvements to Bhyve?

Improvements and fixes to illumos bhyve are almost entirely done in upstream illumos-gate, rather than the Oxide downstream.

Upstreaming those changes into FreeBSD bhyve is a more complicated situation, given that illumos has diverged from upstream over the years due to differing opinions about certain interfaces.

Re: Why Oxide Chose Illumos

#123

Post by bitfilped »

craftkiller wrote: post #79
bitfilped wrote: post #66

Earlier quoted context omitted.

It's been a minute since I messed with bhyve on FreeBSD, but I'm pretty sure you have to switch out the networking stack to something like Netgraph if you intend to use fast networking.

Hmmm I'm not the OP, but I run my personal site on a kubernetes cluster hosted in bhyve VMs running Debian on a FreeBSD machine using netgraph for the networking. I just tested by launching iperf3 on the FreeBSD host and launching an alpine linux pod in the cluster, and I only got ~4Gbit/s. This is surprising to me since netgraph is supposed to be capable of much faster networking but I guess this is going through mu…

Do you know if you're still using if_bridge? I remembered this article from klara that goes a bit more into the details. https://klarasystems.com/articles/using-netgraph-for-freebsd...

Re: Why Oxide Chose Illumos

#124

Post by pmooney »

notaplumber1 wrote: post #102
jclulow wrote: post #40

Earlier quoted context omitted.

Yes, to be clear this is not meant to be a criticism of software quality at OpenBSD! Though I don't necessarily always agree with the leadership style I have big respect for their engineering efforts and obviously as another relatively niche UNIX I feel a certain kinship! That part of the document was also written some years ago, much closer to 2018 when that service got started than now, so it's conceivable that we…

> I will say, though, that single VCPU guests would not have met our immediate needs in the Oxide product! Could Oxide not have helped push multi-vcpu guests out the door by sponsoring one of the main developers working on it, or contributing to development? From a secure design perspective, OpenBSD's vmd is a lot more appealing than bhyve is today. I saw recently that AMD SEV (Secure Encrypted Virtualization) was ad…

> Could Oxide not have helped push multi-vcpu guests out the door by sponsoring one of the main developers working on it, or contributing to development?

Being that vmd's values are aligned with OpenBSD's (security above all else), it is probably not a good fit for what Oxide is trying to achieve. Last I looked at vmd (circa 2019), it was doing essentially all device emulation in userspace. While it makes total sense to keep as much logic as possible out of ring-0 (again, emphasis on security), doing so comes with some substantial performance costs. Heavily used devices, such as the APIC, will incur pretty significant overhead if the emulation requires round trips out to userspace on top of the cost of VM exits.

> I saw recently that AMD SEV (Secure Encrypted Virtualization) was added, which seems compelling for Oxide's AMD based platform. Has Oxide added support for that to their bhyve fork yet?

SEV complicates things like the ability to live-migrate guests between systems.

Re: Why Oxide Chose Illumos

#125

Post by throw0101b »

selykg wrote: post #76
codetrotter wrote: post #20

Earlier quoted context omitted.

> Joyent which was an illumos shop And before that, they used to run FreeBSD. Mentioned for example in this comment by Bryan Cantrill a decade ago: https://news.ycombinator.com/item?id=6254092 > […] Speaking only for us (I work for Joyent), we have deployed hundreds of thousands of zones into production over the years -- and Joyent was running with FreeBSD jails before that […] And I’ve seen some other primary source…

Joyent also merged with TextDrive, which is where the FreeBSD part came from. TextDrive was an early Rails host, and could even do it in a shared hosting environment, which is where I think a lot of the original user base came from (also TextPattern) As I recall they were also the original host of Twitter, which if I recall was Rails back in the day.

> As I recall [Joyent] were also the original host of Twitter, which if I recall was Rails back in the day.

Up until 2008:

* https://web.archive.org/web/20080201142828/http://www.joyeur...

Re: Why Oxide Chose Illumos

#126

Post by timenova »

ComputerGuru wrote: post #101
leoh wrote: post #39

I’d love to use Illumos, but a lack of arm64 support is a non-starter

I don’t mean to downplay the importance for you personally but I do want to clarify that while it might be a non-starter for you, all of arm64 is so new that it’s hardly a non-starter for anyone considering putting it into (traditional) production.

You're right, however I was looking for the same information to maybe try it on a RPi to learn more about Illumos.

Re: Why Oxide Chose Illumos

#127

Post by evandrofisico »

transpute wrote: post #10

> Xen: Large and complicated (by dom0) codebase, discarded for KVM by AMZN 1. Xen Type-1 hypervisor is smaller than KVM/QEMU. 2. Xen "dom0" = Linux/FreeBSD/OpenSolaris. KVM/bhyve also need host OS. 3. AMZN KVM-subset: x86 cpu/mem virt, blk/net via Arm Nitro hardware. 4. bhyve is Type-2. 5. Xen has Type-2 (uXen). 6. Xen dom0/host can be disaggregated (Hyperlaunch), unlike KVM. 7. pKVM (Arm/Android) is smaller than KVM…

I've been using xen in production for at least 18 years, and although there is been some development, it is extremely hard to get actual documentation on how to do things with it.

There is no place documenting how to integrate the Dom0less/Hyperlaunch in a distribution or how to build infrastructure with it, at best you will find a github repo, with the last commit dated 4 years ago, with little to no information on what to do with the code.

Re: Why Oxide Chose Illumos

#128

Post by pclmulqdq »

throw0101b wrote: post #114
pclmulqdq wrote: post #84

Earlier quoted context omitted.

Ceph is sadly not very good at what it does. The big clouds have internal versions of object store that are far better (no single point of failure, much better error recovery story, etc.). ZFS solves a different problem, though. ZFS is a full-featured filesystem. Like Ceph it is also vulnerable to single points of failure.

> The big clouds have internal versions of object store that are far better (no single point of failure, much better error recovery story, etc.). There are different levels of scalability needs. CERN has over a dozen (Ceph) clusters with over 100PB of total data as of 2023: * https://www.youtube.com/watch?v=bl6H888k51w Certainly there are some number of folks that need more than that, but I don't there are many. > Li…

Single-monitor is a common way to run Ceph. On top of that, many cluster configurations cause the whole thing to slow to a crawl when a very small minority of nodes go down. Never mind packet loss, bad switches, and other sorts of weird failure mechanisms. Ceph in general is pretty bad at operating in degraded modes. ZFS and systems like Tectonic (FB) and Colossus (Google) do much better when things aren't going perfectly.

Do you know how many administrators CERN has for its Ceph clusters? Google operates Colossus at ~1000x that size with a team of 20-30 SREs (almost all of whom aren't spending their time doing operations).

Re: Why Oxide Chose Illumos

#129

Post by akira2501 »

TimTheTinker wrote: post #113
akira2501 wrote: post #108

Earlier quoted context omitted.

> it raises the floor on software quality. Languages cannot possibly do this.

I believe TypeScript and Rust are both strong examples of languages that do this (for different reasons and in different ways). It's also possible for a language to raise the ceiling of software quality, and Zig is an excellent example. I'm thinking of "floors" and "ceilings" as the outer bounds of what happens in real, everyday life within particular software ecosystems in terms of software quality. By "quality" I m…

> possible for a language to raise the ceiling of software quality

Cargo is widely recognized as low quality. The thesis fails within it's own standard packaging. It's possible for a language to be used by _more people_ and thus raise the quality _in aggregate_ of produced software but the language itself has no bearing on quality in any objective measure.

> to benefit from a raised ceiling

You're explicitly putting the cart before the horse here. The more reasonable assertion is that it takes good people to get good results regardless of the quality of the tool. Acolytes are uncomfortable saying this because it also destroys the negative case, which is, it would be impossible to write quality software in a previous generation language.

> TigerBeetle[0] is an example

Of a protocol and a particular implementation of that protocol. It has client libraries in multiple languages. This has no bearing on this point.

Re: Why Oxide Chose Illumos

#130

Post by blaerk »

9front wrote: post #112
wmf wrote: post #97

Earlier quoted context omitted.

EC2 migrated off Xen around ten years ago. Only really old instances should be using Xen or Xen emulation.

I'm puzzled by your comment. On an EC2 instance of AL2023 deployed on us-east-1 region this is the output of hostnamectl: [ec2-user][~]$ hostnamectl Static hostname: ip-x-x-x-x.ec2.internal Icon name: computer-vm Chassis: vm Machine ID: ec2d54f27fc534ea74980638ccc33d96 Boot ID: 6caf18b7ed3647819c1985c11f128142 Virtualization: xen Operating System: Amazon Linux 2023.5.20240903 CPE OS Name: cpe:2.3:o:amazon:amazon_linu…

I think some older instance types are still on xen, later types run kvm (code named nitro.. perhaps?). I can’t remember the exact type but last year we ran into some weird issues related to some kernel regression that only affected some instances in our fleet, turns out they where all the same type and apparently ran on xen according to aws support
Post reply on HN