Live data from Hacker News

Helios: A distribution of Illumos powering the Oxide Rack

github.com

151–160 of 268 posts

Re: Helios: A distribution of Illumos powering the Oxide Rack

#151

Earlier quoted context omitted.

OpenStack is pretty smooth sailing these days and I bet you it would be much cheaper to just get 3 FTEs for your OpenStack install than an Oxide rack

Where, exactly, are you getting these 3FTEs qualified to touch production OpenStack infra, for more than a year, where their aggregate cost is less than a rack of equipment?

if you need OpenStack you're not running one rack, but a couple dozen.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#152
post #51

Earlier quoted context omitted.

>including much more of the system within their scope: they are not merely kernels, but also system libraries and commands. giving limited resources of the dev team it may lead to limited support of the system outside of the narrow set of officially supported/certified hardware with that support falling behind on modern hardware, as it happened with Sun, and vendor lock-in as a result into overpriced and low performi…

Well they aren't burdened by having to make their own processors, like Sun had to do, or their own full custom chips in general. They just have to support the selection of hardware they pick, and they have complete oversight of what hardware runs on their racks. So I'm not sure if the sun comparison is relevant here, since they can still pick top of the line hardware. Just not any hardware

Any issues with funding or whatever, and their customers would get locked in on the yesterday's "top of the line hardware" (reminds how Oracle used lawyers to force HP to continue support Itanic). Sun was 50K persons company, and they struggled to support even reasonably wide set of hardware. Vendor lock in is like a Newton law in this industry.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#153
post #39

Earlier quoted context omitted.

Another Xoogler here: any idea what they mean by it's not Linux at the bottom for other providers? Like, surely it's _some_ common OS? Either my binaries wouldn't run or AWS is reimplementing Linux so they can, which seems odd. Or are they just saying that the VM my binary runs on might be some predictable Linux version, but the underlying thing launching the VM could be anything?

Old AWS used to be Xen, Nitro afaik uses customised VMM and I don't recall if it's not a custom OS or hosted on top of something. Azure is Hyper-V underneath IIRC, a custom variant at least (remember Windows Server Nano? IIRC it was the closest you could get to running it), with sometimes weird things like network cards running Linux and integrating with Windows' built-in SDN facility. Rest of the bigger ones is main…

Nitro is supposed to be this super customized version of KVM.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#154
post #56

Earlier quoted context omitted.

> Why would people pay a premium? I would pay a premium just to not have to deal with HPE, DELL, etc

Dell's been nothing but fantastic for us (compute, not storage.)

Dell is a mixed bag depending on how well the individual region you are dealing with is doing overall. Things were great for us, but something changed and now getting good support for hardware failures has been a nightmare of jumping through hoops, time zone handoffs to other teams, and forced on-site techs to replace a stick of ram.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#155

Earlier quoted context omitted.

The best elevator pitch I've heard is "AWS APIs for on-prem datacenters". They make turn-key managed racks that behave just like a commercial cloud would with all the APIs for VM, storage, and network provisioning and integration you'd expect from AWS, except made to deploy in your company's datacenter under your control.

I guess the wildcard is price. AWS's pricing model works kinda at their OMG eyewatering scale - aka all the custom hardware they design is highly cost optimized, but just doing custom hardware has a notable cost. This is easily covered by their scale, to make for their famous margins. [during their low scale times, they did use a good bit of HP/Dell, etc] Oxide seems to be no different (super custom hardware) only ma…

Datacenter costs are weird. The first big cost is having a datacenter. However once you have the space, power, cooling and that part makes sense, then the actual hardware going into it can have a pretty decent premium and still be highly competitive with AWS. It will also depend heavily on what you are doing and producing, if the answer to that is a large amount of data, and it needs to transit out of AWS, suddenly the cost of a pretty large datacenter is really cheap in comparison. AWS egress fees have a markup that will make your accountants panic. From a hardware standpoint, once you need GPU compute or large amounts of RAM, the prices get pretty dumb as well.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#156
post #149

I'm really curious: what kind of workload would companies want to run on a custom Unix that isn't Linux/Mac/BSD? I'm rooting for more mature OS diversity, I just have no idea who the end users would be and what their needs would look like.

ZFS is native on illumos, and the containerization equivalent, etc, is pretty great.

There's a good argument that your servers in the cloud don't need to be on the same OS, as long as you can hire enough talent to work on them.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#157
post #27
post #23

Earlier quoted context omitted.

If you're running in one of the big 3 cloud providers, the bottom-level hypervisors are not-linux. This is equivalent. Are you anti-AWS or anti-Azure for the same reason? This is the substrate upon which you will run any virtualized infrastructure.

Small note, that's not true for Google Cloud, which runs on top of Linux, though modified. Disclaimer: Former Googler, Cloud Support

As I understand it, there's linux running on the Google Cloud hardware but the virtualized networking and storage stacks in Google Cloud are google proprietary and largely bypass linux -- in the case of networking see the "Snap: a Microkernel Approach to Host Networking" paper.

In contrast, it appears that Oxide is committing to open-source the equivalent pieces of their virtualization platform.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#158
post #2

I know they’re ex-Sun, but is there any real technical benefit for choosing not-Linux (for their business value prop)? I know of the technical benefits of illumos over linux, but does that actually matter to the customers who are buying these? Aren’t they opening a whole can of worms for ideology/tradition that won’t sell any more computers? As someone who runs Linux container workloads, the fact that this is fundame…

Keep in mind that Helios is really just an implementation detail of the rack; like Hubris[0], it's not something visible to the user or to applications. (The user of the rack provisions VMs.) As for why an illumos derivative and not something else, we expanded on this a bit in our Q&A when we shipped our first rack[1] -- and we will expand on it again in the (recorded) discussion that we will have later today.[2] [0]…

[deleted]

Re: Helios: A distribution of Illumos powering the Oxide Rack

#159

Earlier quoted context omitted.

I don't expect anyone to see my comments unless they're really looking since I've been shadow banned for many years now - so I appreciate your reply. To be clearer regarding my questions: - What happened to Project X (supposedly coreboot++ for latest AMD CPUs)? It seems dead, despite being more reported on than Oxide's attempts in working with AMD (to achieve the same outcomes, presumably - what's the difference?). L…

So, your comment was not dead when I saw it. This reply was, but apparently now has been vouched for. > What happened to Project X (supposedly coreboot++ for latest AMD CPUs)? I don't recall what you're referring to specifically, maybe this was a thing before I started at Oxide. I do know that we deliberately decided to not go with coreboot. I believe the equivalent component would be phbl[1]. It boots illumos direct…

To see more on "Project X", see the Phoronix article on it. At the very least, it would be resourceful if the Oxide devs had a chat with the Project X devs who have since given up - learnings can be had and time can be saved. And yes, coreboot itself is now untennable, but is also kind of a slang for the a category of deblobbed software.

Re: Helios: A distribution of Illumos powering the Oxide Rack

#160

Earlier quoted context omitted.

Yes! Crucible[0] is on our list of upcoming episodes. We can touch on it tonight, but it's really deserving of its own deep dive! [0] https://github.com/oxidecomputer/crucible

The timing of your podcast is the least convenient thing ever for us poor Europeans. And then the brutal wait the next day until its uploaded. The only thing I miss about Twitter Spaces is that you could listen the morning after.

Yes (hello from Czechia), however there will always be somebody who this is inconvenient for. Also, I have to confess I was at times immersed in other work that I made a few Oxide and Friends live. I might stay up tonight.

I am looking forward to the crucible episode. It sounds like it could be a startup on its own, it wouldn't be the first distributed file/ storage system company.

Post reply on HN