Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Oxide Chose Illumos

rfd.shared.oxide.computer

Post reply on HN
Print view View original

151–160 of 177 posts

Re: Why Oxide Chose Illumos

#151

Post by antongribok »

pclmulqdq wrote: post #128
throw0101b wrote: post #114

Earlier quoted context omitted.

> The big clouds have internal versions of object store that are far better (no single point of failure, much better error recovery story, etc.). There are different levels of scalability needs. CERN has over a dozen (Ceph) clusters with over 100PB of total data as of 2023: * https://www.youtube.com/watch?v=bl6H888k51w Certainly there are some number of folks that need more than that, but I don't there are many. > Li…

Single-monitor is a common way to run Ceph. On top of that, many cluster configurations cause the whole thing to slow to a crawl when a very small minority of nodes go down. Never mind packet loss, bad switches, and other sorts of weird failure mechanisms. Ceph in general is pretty bad at operating in degraded modes. ZFS and systems like Tectonic (FB) and Colossus (Google) do much better when things aren't going perf…

This is complete nonsense. No one running business critical installs of Ceph runs single-monitor.

You can also tell Ceph to use a single disk as your failure domain. No one does that either. Homelabbers maybe, but then why are you comparing such setups with Google?

We run Ceph with a failure domain of an entire rack. We can literally take down (scheduled or unscheduled) an entire rack of 40 servers, and continue to serve critical, latency sensitive applications, with no noticeable performance loss.

We have a Ceph footprint 5x larger than CERN run by a team of 4-5 people.

Re: Why Oxide Chose Illumos

#152

Post by cthalupa »

anonfordays wrote: post #148
dijit wrote: post #141

Earlier quoted context omitted.

Honestly, SMF is superior to SystemD and it’s ironic it came earlier (and, that shows based on the fact that it uses XML as its configuration language.. ick). However, two things are an issue: 1) The CDDL license of SMF makes it difficult to use, or at least that’s what I was told when I asked someone why SMF wasn’t ported to Linux in 2009. 2) SystemD is it now. It’s too complicated to replace and software has become…

>Honestly, SMF is superior to SystemD Maybe 15 years ago, not by a mile now. systemd surpassed SMF years ago and it's not even close now. No one in their right mind would pick SMF over systemd in 2024.

I regularly pick significantly less featured init systems over systemd whenever it is feasible, because systemd and it's related components have caused some of the largest amounts of work for me over the past decade.

I don't really want to litigate the systemd vs. everything else argument, but as someone that has issues with systemd but is not particularly in love with sysvinit derivatives, I wouldn't mind SMF as an alternative.

Re: Why Oxide Chose Illumos

#153

Post by craftkiller »

bitfilped wrote: post #123
craftkiller wrote: post #79

Earlier quoted context omitted.

Hmmm I'm not the OP, but I run my personal site on a kubernetes cluster hosted in bhyve VMs running Debian on a FreeBSD machine using netgraph for the networking. I just tested by launching iperf3 on the FreeBSD host and launching an alpine linux pod in the cluster, and I only got ~4Gbit/s. This is surprising to me since netgraph is supposed to be capable of much faster networking but I guess this is going through mu…

Do you know if you're still using if_bridge? I remembered this article from klara that goes a bit more into the details. https://klarasystems.com/articles/using-netgraph-for-freebsd...

Thanks, but I am not using if_bridge. I am creating a netgraph bridge[0] which is connected to the host via a netgraph eiface. Then on the host, the packet passes to my real physical interface because I have gateway_enable and I have pf perform NAT[1]. It looks like that blog post connected the netgraph bridge directly to the external interface, so my guess is my slowdown is from either pf performing NAT or the packet forwarding from gateway_enable.

    [0] https://code.fizz.buzz/talexander/machine_setup/src/commit/20768edcf69eddae5cf65e30a0bde869f9ddd19b/ansible/roles/bhyve/files/bhyve_netgraph_bridge.bash#L214
    [1] https://code.fizz.buzz/talexander/machine_setup/src/commit/20768edcf69eddae5cf65e30a0bde869f9ddd19b/ansible/roles/firewall/files/mrmanager_pf.conf#L35

Re: Why Oxide Chose Illumos

#154

Post by cLe223 »

TimTheTinker wrote: post #56
bonzini wrote: post #51

Earlier quoted context omitted.

That would be a damn good record though, isn't it? (I am fairly sure that more were found since, but the point is that these are pretty rare). Firecracker, which is written in Rust, had one in 2019: https://www.cve.org/CVERecord?id=CVE-2019-18960 Also QEMU's fuzzing is very sophisticated. Most recent vulnerabilities were found that way rather than by security researchers, which I don't think it's the case for "compet…

You're not wrong, and that is very impressive. There's nothing like well-applied fuzzing to improve security. But I still don't think that makes Oxide's decision or my comment necessarily invalid, if only because of an a priori decision to stick with Rust system-wide -- it raises the floor on software quality.

1. Oxide made an unproven statement ("QEMU is often the subject of bugs affecting its reliability and security.")

2. OP (bonzini) has given specific and valid arguments that that statement is wrong.

3. You're not answering to that specific arguments, but defending Rust and bashing C++ generally without giving any prove.

4. bonzini again provides specific arguments that your generalization is not correct in that context. That despite Firecracker being written in Rust it had a security issue.

5. You still insist without given any solid argument. You just insist that Rust is superior. Not helpful in any discussion. Think about it.

Re: Why Oxide Chose Illumos

#155

Post by dijit »

anonfordays wrote: post #148
dijit wrote: post #141

Earlier quoted context omitted.

Honestly, SMF is superior to SystemD and it’s ironic it came earlier (and, that shows based on the fact that it uses XML as its configuration language.. ick). However, two things are an issue: 1) The CDDL license of SMF makes it difficult to use, or at least that’s what I was told when I asked someone why SMF wasn’t ported to Linux in 2009. 2) SystemD is it now. It’s too complicated to replace and software has become…

>Honestly, SMF is superior to SystemD Maybe 15 years ago, not by a mile now. systemd surpassed SMF years ago and it's not even close now. No one in their right mind would pick SMF over systemd in 2024.

The fact that its less opinionated about logging and networking and doesnt ever force any reload of itself are all reasonable reasons to prefer it.

You don’t lose socket activation or supervison. SMF is designed to help work in the event of hardware failure too, which systemd definitely cant handle.

Re: Why Oxide Chose Illumos

#156

Post by otabdeveloper4 »

sophacles wrote: post #134
otabdeveloper4 wrote: post #74

Earlier quoted context omitted.

[flagged]

Why is it the only people who say this at all are people saying it sarcastically or quoting fictional strawmen (and can never seem to provide evidence of it being said in earnest)?

Grandparent comment said it in earnest, do keep up.

Re: Why Oxide Chose Illumos

#157

Post by mwcampbell »

jclulow wrote: post #110
stonogo wrote: post #75

Earlier quoted context omitted.

So at no point did anyone even suspect that Illumos was under consideration because it's been corporate leadership's pet project for decades? That seems like a wild thing to omit from the "RFD" process. Or were some topics not open to the "RFD" process?

We are trying to build a business here. The goal is to sell racks and racks of computers to people, not build a menagerie of curiosities and fund personal projects. Everything we've written here is real, at least from our point of view. If we didn't think it would work, why would we throw our own business, and equity, and so on, away? The reason I continue to invest myself, if nothing else, in illumos, is because I g…

Furthermore, your team have already demonstrated that you can reevaluate things that you had strong opinions on, and come to a different conclusion. I'm thinking of the decision to exclusively run hardware-virtualized VMs rather than LX-branded zones, as we discussed on Oxide and Friends before.

I don't think working at Oxide would be for me, but I respect the team's values and process.

Re: Why Oxide Chose Illumos

#158

Post by jonstewart »

parasubvert wrote: post #63
jonstewart wrote: post #49

Illumos makes sense as a host OS—it’s capable, they know it, they can make sure it works well on their hardware, and virtualization means users don’t need that much familiarity with it. If I were Oxide, though, I’d be sprinting to seamless VMWare support. Broadcom has turned into a modern-day Oracle (but dumber??) and many customers will migrate in the next two years. Even if those legacy VMs aren’t “hyperscale”, the…

Oracle is a $53 billion company, and never had a mass exodus, just less greenfield deployments. Broadcom also isn't all that dumb, VMware was fat and lazy and customers were coddled for a very long time. They've made a bet that it's sticky. The competition isn't as weak as they thought, that's true, but it will take 5+ years to catch up, not 2 years, in general. Broadcom was betting on it taking 10 years: plenty of t…

I don’t disagree much. Still, there’s a sudden weakening of the main incumbent in the on-prem virtualization market… and that is _all_ Oxide does. It will be interesting to see whether Oxide can convert some VMWare customers.

Re: Why Oxide Chose Illumos

#159

Post by meepmorp »

jonstewart wrote: post #158
parasubvert wrote: post #63

Earlier quoted context omitted.

Oracle is a $53 billion company, and never had a mass exodus, just less greenfield deployments. Broadcom also isn't all that dumb, VMware was fat and lazy and customers were coddled for a very long time. They've made a bet that it's sticky. The competition isn't as weak as they thought, that's true, but it will take 5+ years to catch up, not 2 years, in general. Broadcom was betting on it taking 10 years: plenty of t…

I don’t disagree much. Still, there’s a sudden weakening of the main incumbent in the on-prem virtualization market… and that is _all_ Oxide does. It will be interesting to see whether Oxide can convert some VMWare customers.

Maybe some, but the main difference between Oxide and the others is that they sell an integrated platform, so if a customer is just looking to replace the VM software, that seems like a harder sell.

Re: Why Oxide Chose Illumos

#160

Post by throw0101b »

pclmulqdq wrote: post #128
throw0101b wrote: post #114

Earlier quoted context omitted.

> The big clouds have internal versions of object store that are far better (no single point of failure, much better error recovery story, etc.). There are different levels of scalability needs. CERN has over a dozen (Ceph) clusters with over 100PB of total data as of 2023: * https://www.youtube.com/watch?v=bl6H888k51w Certainly there are some number of folks that need more than that, but I don't there are many. > Li…

Single-monitor is a common way to run Ceph. On top of that, many cluster configurations cause the whole thing to slow to a crawl when a very small minority of nodes go down. Never mind packet loss, bad switches, and other sorts of weird failure mechanisms. Ceph in general is pretty bad at operating in degraded modes. ZFS and systems like Tectonic (FB) and Colossus (Google) do much better when things aren't going perf…

> Single-monitor is a common way to run Ceph.

What?

> A Ceph cluster must contain a minimum of three running monitors in order to be both redundant and highly-available.

* https://docs.ceph.com/en/latest/glossary/#term-Ceph-Monitor

> Our Configuring ceph section provides a trivial Ceph configuration file that provides for one monitor in the test cluster. A cluster will run fine with a single monitor; however, a single monitor is a single-point-of-failure. To ensure high availability in a production Ceph Storage Cluster, you should run Ceph with multiple monitors so that the failure of a single monitor WILL NOT bring down your entire cluster.

* https://docs.ceph.com/en/latest/rados/configuration/mon-conf...

Post reply on HN