Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Helios: A distribution of Illumos powering the Oxide Rack

github.com

Post reply on HN
Print view View original

81–90 of 268 posts

Re: Helios: A distribution of Illumos powering the Oxide Rack

#81

Post by amluto »

wmf wrote: post #40
amluto wrote: post #36

Earlier quoted context omitted.

Especially since Oxide has a big fancy firmware stack. I would expect this stack to be able to do an excellent job of securely allocating bare-metal (i.e. VMX root on x86 or EL2 if Oxide ever goes ARM) resources. This would allow workloads on Oxide to run their own VMs, to safely use PCIe devices without dealing with interrupt redirection, etc.

I'm not affiliated with Oxide but I don't think you can put Crucible and VPC/OPTE in firmware. Without a DPU those components have to run in the hypervisor.

Possibly not.

But I do wonder why cloud and cloud-like systems aren’t more aggressive about splitting the infrastructure and tenant portions of each server into different pieces of hardware, e.g. DPU. A DPU could look DPU could look like a PCIe target exposing NVMe and a NIC, for example.

Obviously this would be an even more custom design than Oxide currently has, but Oxide doesn’t seem particularly shy about such things.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#82

Post by kaliszad »

bcantrill wrote: post #9
sneak wrote: post #2

I know they’re ex-Sun, but is there any real technical benefit for choosing not-Linux (for their business value prop)? I know of the technical benefits of illumos over linux, but does that actually matter to the customers who are buying these? Aren’t they opening a whole can of worms for ideology/tradition that won’t sell any more computers? As someone who runs Linux container workloads, the fact that this is fundame…

Keep in mind that Helios is really just an implementation detail of the rack; like Hubris[0], it's not something visible to the user or to applications. (The user of the rack provisions VMs.) As for why an illumos derivative and not something else, we expanded on this a bit in our Q&A when we shipped our first rack[1] -- and we will expand on it again in the (recorded) discussion that we will have later today.[2] [0]…

Perhaps you could talk a bit about the distributed storage based on Crucible with ZFS as the backing storage tonight. I would really love to hear some of the details and challenges there.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#84

Post by jeffbee »

stonogo wrote: post #20
jeffbee wrote: post #13

Earlier quoted context omitted.

Do you have the same gut reaction to ESXi?

I sure do. We've finally got to a place where we don't need weird hardware tricks to containerize workloads -- this is why a lot of shops pursue docker-like ops for production. When I buy hardware, long-term maintenance is a factor, and when my whole operations fleet relies on ESX, or in this case a Solaris fork, I'm now beholden to one company for support at that layer. Buying a rack of Supermicro gear and running R…

I can somewhat see your point, but in my experience you can't rely on RHEL or whatever vendor Linux to correctly bring up random OEM hardware. You will slowly discover all of the quirks, like it didn't initialize the platform EDAC the way you expected, or it didn't resolve some weird IRQ issue, etc. Nothing about my experience leads me to believe Linux will JFW on a given box, so I don't feel like Linux has an advantage in this regard, or that niche operating systems have a disadvantage. Certainly I feel like a first-party OS from the hardware vendor is going to have a lot of advantages.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#85

Post by Hackbraten »

lifeisstillgood wrote: post #79

I would be interested in how did you first hear of Oxide . I somehow landed on their podcast because it covered . The podcast is for me amazeballs marketing - it does everything but sell their product (might be a good idea to add a pitch in for each out-tro!) I mean they talk about it, like “we had such a tough time getting the compiler to do something something and then veer off to discuss back in the day stories. A…

Was following @jessfraz on Twitter back then, so I got word of Oxide when they first announced it there.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#86

Post by steveklabnik »

moondev wrote: post #74
sneak wrote: post #2

I know they’re ex-Sun, but is there any real technical benefit for choosing not-Linux (for their business value prop)? I know of the technical benefits of illumos over linux, but does that actually matter to the customers who are buying these? Aren’t they opening a whole can of worms for ideology/tradition that won’t sell any more computers? As someone who runs Linux container workloads, the fact that this is fundame…

The main drawbacks to me are 1. No support for nested virtualization, so running a vm inside your vm is not available. This prevents use of projects such as kubevirt or firecracker on a Linux guest, and WSL2 on a Windows guest. 2. No GPU support If the base hypervisor was Linux, it would be way more capable for users it seems. I also wonder if internally Linux is used for development of the platform itself so they ca…

> I also wonder if internally Linux is used for development of the platform itself

Developers at Oxide work on whatever platform they'd like, as long as they can do their work. I will say I am in the minority as a Windows user though, most are on some form of Unix.

> so they can create "virtual" racks to dogfood the product without full blown physical racks.

So one of the reasons why Rust is such an advantage for us is its strong cross-platform support: you can run a simulated version of the control plane on Mac, Linux, and Illumos, without a physical rack. The non-simulated version must run on Helios. [1]

That said we do have a rack in the office (literally named dogfood) that employees can use for various things if they wish.

1: https://github.com/oxidecomputer/omicron?tab=readme-ov-file#...

Re: Helios: A distribution of Illumos powering the Oxide Rack

#87

Post by panick21_ »

throwawaaarrgh wrote: post #54
refulgentis wrote: post #29

Earlier quoted context omitted.

Can you get me excited? I spent 20 seconds browsing the homepage and walked away with "so the idea is vertical integration for on-premise server purchases? On custom OS? Why? Why would people pay a premium?" But immediately got myself to "what does a server OS do anyway, doesn't it just launch VMs? You don't need Linux, just the ability to launch Linux VMs" Tell me more? :)

It's a mainframe. If you can't get excited for mainframes it'll be hard to be excited about this. IllumOS is the OS/360 to Oxide's System/360. (It won't get that popular but it's a fair enough comparison for illustrative purposes)

Except that it use the same standard CPU as commodity machine. Doesn't have much of the extra reliability stuff. It can go from vertical to horizontal scaling. The OS is open source Unix. And yeah its not like a mainframe at all really.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#88

Post by bcantrill »

kaliszad wrote: post #82
bcantrill wrote: post #9

Earlier quoted context omitted.

Keep in mind that Helios is really just an implementation detail of the rack; like Hubris[0], it's not something visible to the user or to applications. (The user of the rack provisions VMs.) As for why an illumos derivative and not something else, we expanded on this a bit in our Q&A when we shipped our first rack[1] -- and we will expand on it again in the (recorded) discussion that we will have later today.[2] [0]…

Perhaps you could talk a bit about the distributed storage based on Crucible with ZFS as the backing storage tonight. I would really love to hear some of the details and challenges there.

Yes! Crucible[0] is on our list of upcoming episodes. We can touch on it tonight, but it's really deserving of its own deep dive!

[0] https://github.com/oxidecomputer/crucible

Re: Helios: A distribution of Illumos powering the Oxide Rack

#89

Post by _rs »

EvanAnderson wrote: post #43
milon wrote: post #24

I'm glad this is out, i'm going to deploy this locally and learn as much about it as possible. Oxide is pretty much the company I dream to work at, both for the tech stack, plus the people working there. Thank you Oxide team!

I'm excited to see how this compares to SmartOS. I'm pretty heavily invested in SmartOS in my personal infrastructure but its future, post-Joyent acquisition, has been worrying me. I really wish I did work for an org big enough to use Oxide's gear. Not having to futz around with bogus IBM PC AT-type compatibility edifice, janky BMCs and iDRACs, hardware RAID controllers, etc, would be so unbelievably nice.

I had been using SmartOS for a long time but finally had to bite the bullet and give up. I ended up deciding on Proxmox on a ZFS root and am quite happy with it.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#90

Post by Jtsummers »

wmf wrote: post #70
throwawaaarrgh wrote: post #60

Earlier quoted context omitted.

A team should always pick the tools they are most familiar with. They will always have better results with that, than trying to use something they understand less. With this in mind, using their own stack is a perfectly adequate choice. Factors outside their team will determine if that works out in the long term.

A handful of the team are more familiar with Illumos and the next hundred people they hire after that will be more familiar with Linux.

If your hiring decisions are always based on what people are currently familiar with, you'll always be stuck in the past. You may not even be able to use present day tooling and systems because they could be too new to hire people for.

You're much better off hiring people who are capable of learning, and then giving them the opportunities to learn and advance their knowledge and skills.

Post reply on HN