Helios: A distribution of Illumos powering the Oxide Rack • Hacker News Board

Re: Helios: A distribution of Illumos powering the Oxide Rack

#51

Post by trhway » Mon, Jan 29, 2024, 7:35 PM UTC

bcantrill wrote: post #34

StillBored wrote: post #26

Earlier quoted context omitted.

Linux is a nightmare in the embedded/appliance space because one ends up just having platform engineers who spend their day fixing problems with the latest kernels, drivers, core libraries, etc, that the actual application depends on. Or one goes the route of 99% of the IoT/etc vendors, and never update the base OS and pray that there aren't any active exploits targeting it. This is why a lot of medium-sized companie…

This is a really, really good point -- and is a result of the model of Linux being only a kernel (and not system libraries, commands, etc.). It means that any real use of Linux is not merely signing up for kernel maintenance (which itself can be arduous) but also must make decisions around every other aspect of the system (each with its own communities, release management, etc.). This act is the act of creating a dis…

>including much more of the system within their scope: they are not merely kernels, but also system libraries and commands.

giving limited resources of the dev team it may lead to limited support of the system outside of the narrow set of officially supported/certified hardware with that support falling behind on modern hardware, as it happened with Sun, and vendor lock-in as a result into overpriced and low performing hardware.

There is a reason that back then at Solaris dev there was a joke about embedding Linux kernel as a universal driver for Solaris kernel in order to get reasonable support for the hardware around.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#52

Post by tptacek » Mon, Jan 29, 2024, 7:35 PM UTC

greggyb wrote: post #23

sneak wrote: post #2

I know they’re ex-Sun, but is there any real technical benefit for choosing not-Linux (for their business value prop)? I know of the technical benefits of illumos over linux, but does that actually matter to the customers who are buying these? Aren’t they opening a whole can of worms for ideology/tradition that won’t sell any more computers? As someone who runs Linux container workloads, the fact that this is fundame…

If you're running in one of the big 3 cloud providers, the bottom-level hypervisors are not-linux. This is equivalent. Are you anti-AWS or anti-Azure for the same reason? This is the substrate upon which you will run any virtualized infrastructure.

I don't about EC2 but Lambda and Fargate are presumably Firecracker, which is Linux KVM.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#53

Post by bpye » Mon, Jan 29, 2024, 7:36 PM UTC

bcantrill wrote: post #16

tonyarkles wrote: post #11

Earlier quoted context omitted.

As far as I recall it's not a VM. They run in "LX Branded Zones" which does require a Linux userland so that the binaries can find their libraries etc but Zones are more like "better cgroups than cgroups, a decade earlier" than VMs.

No, it's a VM, running a bhyve-based hypervisor, Propolis.[0] LX branded zones were/are great -- but for absolute fidelity one really needs VMs. [0] https://github.com/oxidecomputer/propolis

Do you have a solution for running containers (Kubernetes, etc)? Are you spinning up a Linux VM to run the containers in there, doing VM per container, or something else?

Re: Helios: A distribution of Illumos powering the Oxide Rack

#54

Post by throwawaaarrgh » Mon, Jan 29, 2024, 7:36 PM UTC

refulgentis wrote: post #29

milon wrote: post #24

I'm glad this is out, i'm going to deploy this locally and learn as much about it as possible. Oxide is pretty much the company I dream to work at, both for the tech stack, plus the people working there. Thank you Oxide team!

Can you get me excited? I spent 20 seconds browsing the homepage and walked away with "so the idea is vertical integration for on-premise server purchases? On custom OS? Why? Why would people pay a premium?" But immediately got myself to "what does a server OS do anyway, doesn't it just launch VMs? You don't need Linux, just the ability to launch Linux VMs" Tell me more? :)

It's a mainframe. If you can't get excited for mainframes it'll be hard to be excited about this.

IllumOS is the OS/360 to Oxide's System/360. (It won't get that popular but it's a fair enough comparison for illustrative purposes)

Re: Helios: A distribution of Illumos powering the Oxide Rack

#55

Post by mustache_kimono » Mon, Jan 29, 2024, 7:37 PM UTC

refulgentis wrote: post #29

milon wrote: post #24

I'm glad this is out, i'm going to deploy this locally and learn as much about it as possible. Oxide is pretty much the company I dream to work at, both for the tech stack, plus the people working there. Thank you Oxide team!

Can you get me excited? I spent 20 seconds browsing the homepage and walked away with "so the idea is vertical integration for on-premise server purchases? On custom OS? Why? Why would people pay a premium?" But immediately got myself to "what does a server OS do anyway, doesn't it just launch VMs? You don't need Linux, just the ability to launch Linux VMs" Tell me more? :)

> so the idea is vertical integration for on-premise server purchases? On custom OS? Why? Why would people pay a premium?

As I understand it, re: vertical integration, the term is actually "hyperconverged". Here, that means it's designed at the level of the rack. Like -- there aren't per compute unit redundant power supplies. There is one DC bus bar conversion for the rack. There is an integrated switch designed by Oxide. There is one company to blame when anything inside the box isn't working.

In addition, the pitch is they're using open source Rust-based firmware for many of the core components (the base board management controller/service processor, and root of trust), and the box presents a cloud like API to provision.

If the problem is: I'm running lots of VMs in the cloud. I'm used to the cloud. I like the way the cloud works, but I need an on-prem cloud, this makes that much easier than other DIY ways to achieve (OMG we need a team of people to build us a cloud...).

Re: Helios: A distribution of Illumos powering the Oxide Rack

#56

Post by lijok » Mon, Jan 29, 2024, 7:40 PM UTC

refulgentis wrote: post #29

milon wrote: post #24

I'm glad this is out, i'm going to deploy this locally and learn as much about it as possible. Oxide is pretty much the company I dream to work at, both for the tech stack, plus the people working there. Thank you Oxide team!

Can you get me excited? I spent 20 seconds browsing the homepage and walked away with "so the idea is vertical integration for on-premise server purchases? On custom OS? Why? Why would people pay a premium?" But immediately got myself to "what does a server OS do anyway, doesn't it just launch VMs? You don't need Linux, just the ability to launch Linux VMs" Tell me more? :)

> Why would people pay a premium?

I would pay a premium just to not have to deal with HPE, DELL, etc

Re: Helios: A distribution of Illumos powering the Oxide Rack

#57

Post by mardifoufs » Mon, Jan 29, 2024, 7:41 PM UTC

trhway wrote: post #51

bcantrill wrote: post #34

Earlier quoted context omitted.

This is a really, really good point -- and is a result of the model of Linux being only a kernel (and not system libraries, commands, etc.). It means that any real use of Linux is not merely signing up for kernel maintenance (which itself can be arduous) but also must make decisions around every other aspect of the system (each with its own communities, release management, etc.). This act is the act of creating a dis…

>including much more of the system within their scope: they are not merely kernels, but also system libraries and commands. giving limited resources of the dev team it may lead to limited support of the system outside of the narrow set of officially supported/certified hardware with that support falling behind on modern hardware, as it happened with Sun, and vendor lock-in as a result into overpriced and low performi…

Well they aren't burdened by having to make their own processors, like Sun had to do, or their own full custom chips in general. They just have to support the selection of hardware they pick, and they have complete oversight of what hardware runs on their racks. So I'm not sure if the sun comparison is relevant here, since they can still pick top of the line hardware. Just not any hardware

Re: Helios: A distribution of Illumos powering the Oxide Rack

#58

Post by lijok » Mon, Jan 29, 2024, 7:42 PM UTC

SteveNuts wrote: post #47

refulgentis wrote: post #29

Earlier quoted context omitted.

Can you get me excited? I spent 20 seconds browsing the homepage and walked away with "so the idea is vertical integration for on-premise server purchases? On custom OS? Why? Why would people pay a premium?" But immediately got myself to "what does a server OS do anyway, doesn't it just launch VMs? You don't need Linux, just the ability to launch Linux VMs" Tell me more? :)

It seems like the folks on HN tend to think the world runs on AWS (I'm not trying to say they don't have a huge market share), but many huge enterprises still run their own datacenters and buy ungodly amounts of hardware. The products that are on the market for an AWS-like experience on-prem are still fairly horrible. A lot of times the solutions are collaborations between vendors, which makes support a huge pain (fi…

Wouldn't a "full AWS experience in their datacenter" be AWS Outpost?

Re: Helios: A distribution of Illumos powering the Oxide Rack

#59

Post by steveklabnik » Mon, Jan 29, 2024, 7:43 PM UTC

littlestymaar wrote: post #25

steveklabnik wrote: post #19

Earlier quoted context omitted.

As I elaborated above, bcantrill did not decree that we must use illumos. Technical decisions are not handed down from above at Oxide.

I saw your comment[1] after I wrote mine, but I'm not saying that he's forcing you guys to use it (that would not a good way of being a CTO at a start-up…), but that doesn't prevent him from advocating for solutions he believes in. Would you say that Oxide would have chosen Illumos if he wasn't part of the company? [1]: https://news.ycombinator.com/item?id=39180706

> Would you say that Oxide would have chosen Illumos if he wasn't part of the company?

I don't know how to respond to this question, because to me it reads like "if things were completely different, what would they be like?" I have no idea if you could even argue that a company could be the same company with different founders.

What I can say is that this line of questioning still makes me feel like you're implying that this choice was made simply based on preference. It was not. I am employee #17 at Oxide, and the decision still wasn't made by the time I joined. But again, the choice was made based on a number of technical factors. The RFD wasn't even authored by Bryan, but instead by four other folks at Oxide. We all (well, everyone who wanted to, I say "we" because I in fact did) wrote out the pros and cons of both, and we weighed it like we would weigh any technical decision: that is, not as a battle of sports teams, but as a "hey we need to drive some screws: should we use a screwdriver, a hammer, or something else?" sort of nuts-and-bolts engineering decision.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#60

Post by throwawaaarrgh » Mon, Jan 29, 2024, 7:43 PM UTC

steveklabnik wrote: post #14

sneak wrote: post #2

I know they’re ex-Sun, but is there any real technical benefit for choosing not-Linux (for their business value prop)? I know of the technical benefits of illumos over linux, but does that actually matter to the customers who are buying these? Aren’t they opening a whole can of worms for ideology/tradition that won’t sell any more computers? As someone who runs Linux container workloads, the fact that this is fundame…

> does that actually matter to the customers who are buying these? It's not like we specifically say "oh btw there's illumos inside and that's why you should buy the rack." It's not a customer-facing detail of the product. I'm sure most will never even know that this is the case. What customers do care about is that the rack is efficient, reliable, suits their needs, etc. Choosing illumos instead of Linux here is a c…

A team should always pick the tools they are most familiar with. They will always have better results with that, than trying to use something they understand less. With this in mind, using their own stack is a perfectly adequate choice. Factors outside their team will determine if that works out in the long term.