Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Oxide Chose Illumos

rfd.shared.oxide.computer

Post reply on HN
Print view View original

111–120 of 177 posts

Re: Why Oxide Chose Illumos

#111

Post by seabrookmx »

ComputerGuru wrote: post #100
anonfordays wrote: post #88

Earlier quoted context omitted.

ZFS and Ceph is apples to oranges. ZFS is scoped to a single host, Ceph can span data centers.

It’s very possible to run a light/small layer on top of ZFS (either userspace daemon or via FUSE) to get you most of the way to scaling ZFS-backed object storage within or across data centers depending on what specific availability metrics you need.

What does this light/small layer look like?

In my experience you need something like GlusterFS which I wouldn't call "light".

Re: Why Oxide Chose Illumos

#112

Post by 9front »

wmf wrote: post #97
9front wrote: post #91

Earlier quoted context omitted.

EC2 instances are using the Xen hypervisor. At least that's what reported by hostnamectl.

EC2 migrated off Xen around ten years ago. Only really old instances should be using Xen or Xen emulation.

I'm puzzled by your comment. On an EC2 instance of AL2023 deployed on us-east-1 region this is the output of hostnamectl:

  [ec2-user][~]$ hostnamectl
   Static hostname: ip-x-x-x-x.ec2.internal
         Icon name: computer-vm
           Chassis: vm 
        Machine ID: ec2d54f27fc534ea74980638ccc33d96
           Boot ID: 6caf18b7ed3647819c1985c11f128142
    Virtualization: xen
  Operating System: Amazon Linux 2023.5.20240903
       CPE OS Name: cpe:2.3:o:amazon:amazon_linux:2023
            Kernel: Linux 6.1.106-116.188.amzn2023.x86_64
      Architecture: x86-64
   Hardware Vendor: Xen
    Hardware Model: HVM domU
  Firmware Version: 4.11.amazon

Re: Why Oxide Chose Illumos

#113

Post by TimTheTinker »

akira2501 wrote: post #108
TimTheTinker wrote: post #56

Earlier quoted context omitted.

You're not wrong, and that is very impressive. There's nothing like well-applied fuzzing to improve security. But I still don't think that makes Oxide's decision or my comment necessarily invalid, if only because of an a priori decision to stick with Rust system-wide -- it raises the floor on software quality.

> it raises the floor on software quality. Languages cannot possibly do this.

I believe TypeScript and Rust are both strong examples of languages that do this (for different reasons and in different ways).

It's also possible for a language to raise the ceiling of software quality, and Zig is an excellent example.

I'm thinking of "floors" and "ceilings" as the outer bounds of what happens in real, everyday life within particular software ecosystems in terms of software quality. By "quality" I mean all of capabilities, performance, and absence of problems.

It takes a team of great engineers (and management willing to take a risk) to benefit from a raised ceiling. TigerBeetle[0] is an example of what happens when you pair a great team, great research, and a high-ceiling language.

[0] https://tigerbeetle.com/

Re: Why Oxide Chose Illumos

#114

Post by throw0101b »

pclmulqdq wrote: post #84
throw0101b wrote: post #46

Somewhat related, they discussed why they chose to use ZFS for their storage backend as opposed to (say) Ceph in a podcast episode: * https://www.youtube.com/watch?v=UvEKSqBBcZw Certainly they already had experience with ZFS (as it is built into Illumos/Solaris), but as it was told to them by someone they trusted who ran a lot of Ceph: " Ceph is operated, not shipped [like ZFS] ". There's more care-and-feeding requir…

Ceph is sadly not very good at what it does. The big clouds have internal versions of object store that are far better (no single point of failure, much better error recovery story, etc.). ZFS solves a different problem, though. ZFS is a full-featured filesystem. Like Ceph it is also vulnerable to single points of failure.

> The big clouds have internal versions of object store that are far better (no single point of failure, much better error recovery story, etc.).

There are different levels of scalability needs. CERN has over a dozen (Ceph) clusters with over 100PB of total data as of 2023:

* https://www.youtube.com/watch?v=bl6H888k51w

Certainly there are some number of folks that need more than that, but I don't there are many.

> Like Ceph it is also vulnerable to single points of failure.

The SPOF for ZFS is the host (unless you replicate, e.g., zfs send).

What is SPOF of Ceph? You can have multiple monitors, managers, and MDSes.

Re: Why Oxide Chose Illumos

#115

Post by throw0101b »

anonfordays wrote: post #88
throw0101b wrote: post #46

Somewhat related, they discussed why they chose to use ZFS for their storage backend as opposed to (say) Ceph in a podcast episode: * https://www.youtube.com/watch?v=UvEKSqBBcZw Certainly they already had experience with ZFS (as it is built into Illumos/Solaris), but as it was told to them by someone they trusted who ran a lot of Ceph: " Ceph is operated, not shipped [like ZFS] ". There's more care-and-feeding requir…

ZFS and Ceph is apples to oranges. ZFS is scoped to a single host, Ceph can span data centers.

> ZFS and Ceph is apples to oranges.

Oxide is shipping an on-prem 'cloud appliance'. From the customer's/user's perspective of calling an API asking for storage, it does not matter what the backend is—apple or orange—as long as "fruit" (i.e., a logical bag of a certain size to hold bits) is the result that they get back.

Re: Why Oxide Chose Illumos

#116

Post by wmf »

9front wrote: post #112
wmf wrote: post #97

Earlier quoted context omitted.

EC2 migrated off Xen around ten years ago. Only really old instances should be using Xen or Xen emulation.

I'm puzzled by your comment. On an EC2 instance of AL2023 deployed on us-east-1 region this is the output of hostnamectl: [ec2-user][~]$ hostnamectl Static hostname: ip-x-x-x-x.ec2.internal Icon name: computer-vm Chassis: vm Machine ID: ec2d54f27fc534ea74980638ccc33d96 Boot ID: 6caf18b7ed3647819c1985c11f128142 Virtualization: xen Operating System: Amazon Linux 2023.5.20240903 CPE OS Name: cpe:2.3:o:amazon:amazon_linu…

What instance type is it?

Re: Why Oxide Chose Illumos

#117

Post by _zagj »

Ctrl+f Cantrill >Phrase not found

Bryan Cantrill, ex-Sun dev, ex-Joyent CTO, now CTO of Oxide, is the reason they chose Illumos. Oxide is primarily an attempt to give Solaris (albeit Rustified) a second life, similar to Joyent before. The company even cites Sun co-founder Scott McNealy for its principles:

https://oxide.computer/principles

>"Kick butt, have fun, don't cheat, love our customers and change computing forever."

>If this sounds familiar, it's because it's essentially Scott McNealy's coda for Sun Microsystems.

Re: Why Oxide Chose Illumos

#118

Post by anonfordays »

throw0101b wrote: post #115
anonfordays wrote: post #88

Earlier quoted context omitted.

ZFS and Ceph is apples to oranges. ZFS is scoped to a single host, Ceph can span data centers.

> ZFS and Ceph is apples to oranges. Oxide is shipping an on-prem 'cloud appliance'. From the customer's/user's perspective of calling an API asking for storage, it does not matter what the backend is—apple or orange—as long as "fruit" (i.e., a logical bag of a certain size to hold bits) is the result that they get back.

Yes, it could be NTFS behind the scenes, but this is still an apples to oranges comparison because the storage service Oxide created is Crucible[0], not ZFS. Crucible is more of an apples to apples comparison with Ceph.

[0] https://github.com/oxidecomputer/crucible

Re: Why Oxide Chose Illumos

#119

Post by hinkley »

bonzini wrote: post #5

> QEMU is often the subject of bugs affecting its reliability and security. {{citation needed}}? When I ran the numbers in 2019, there hadn't been guest exploitable vulnerabilities that affected devices normally used for IaaS for 3 years. Pretty much every cloud outside the big three (AWS, GCE, Azure) runs on QEMU. Here's a talk I gave about it that includes that analysis: slides - https://kvm-forum.qemu.org/2019/kvm…

If they are being precise, then “reliability and security” means something different than “security and reliability”.

How many reliability bugs has QEMU experienced in this time?

The man power to go on site and deal with in the field problems could be crippling. You often pick the boring problems for this reason. High touch is super expensive. Just look at Ferrari.

Re: Why Oxide Chose Illumos

#120

Post by bonzini »

9front wrote: post #112
wmf wrote: post #97

Earlier quoted context omitted.

EC2 migrated off Xen around ten years ago. Only really old instances should be using Xen or Xen emulation.

I'm puzzled by your comment. On an EC2 instance of AL2023 deployed on us-east-1 region this is the output of hostnamectl: [ec2-user][~]$ hostnamectl Static hostname: ip-x-x-x-x.ec2.internal Icon name: computer-vm Chassis: vm Machine ID: ec2d54f27fc534ea74980638ccc33d96 Boot ID: 6caf18b7ed3647819c1985c11f128142 Virtualization: xen Operating System: Amazon Linux 2023.5.20240903 CPE OS Name: cpe:2.3:o:amazon:amazon_linu…

KVM can emulate the Xen hypercall interface. Amazon is not using Xen anymore.
Post reply on HN