What's new here
I think these docs were just published.
These have been online for a few months actually, though the release notes have only just been added.
Oxide Computer: Docs
111–120 of 160 posts
Re: Oxide Computer: Docs
#112Everything about Oxide's gear sounds like fun. I imagine it must be a bit like what working with minicomputers in 70s thru the 90s was like. I did a little work in the late 90s with Alpha-based machines. I was impressed at those machines didn't seem like the hack-job crap that PC-based stuff was (with simulated chips from the early 1980's hiding out in dark corners because "compatibility") and still is today. I'm bet…
I really wish they'd do a tour of it, hardware & software. They've gotta be proud as hell of what they've pulled off, it boggles my mind that they're not more eager to show it off.
Given it was only relatively recently they were talking about getting their first deployment out the door, I'd suspect that staying mostly quiet is strategically wise in terms of avoiding having more prospective customers come in the door at once than they can (as yet) service to the standard they want to.
So "they probably really really -want- to show it off, but are showing remarkable restraint for good reason" seems at the very least plausible to me.
Re: Oxide Computer: Docs
#113Earlier quoted context omitted.
> Commodity servers have the same specs. Probably not a coincidence. It would be interesting to know which ODM they partnered with for the hardware. I've done some work with SuperMicro in the past. Some of their boards come with extensive headers and customization options right out of the box. They're also happy to work on board level customizations with the right contracts in place.
We didn't work with an ODM: the ODMs were unwilling to contemplate some of the most basic things we needed (e.g., replacing the BMC with a much lighter weight service processor, having a true root-of-trust, etc.) -- let alone the more things we wanted to do (e.g., our own switch). The compute sled and the switch are both of our own design and look nothing like what you'll find from an ODM; if you're curious in the de…
Please consider generating some sort of automated transcript from these.
I'd hope your target audience would understand the limitations of such a thing, and I'm probably not the only person who'd rather read than listen even with the obvious caveats.
(these days automated transcripts seem to be no harder to mentally fix up the errors in as I read them than "somebody typing fast on a software keyboard and suffering the inevitable tyop and autocorrupt related issues" is, though of course others' mileage may vary)
Re: Oxide Computer: Docs
#114I’d love to see someone provide a turnkey managed bare metal container platform, complete with L4 / L7 routing. I haven’t heard if oxide has a container play, and I suspect it may require virtualization based on their choice of host OS.
By "L4 / L7 routing" do you mean specifically something that works like Amazon's Application Load Balancer and Network Load Balancer for Kubernetes? And by "turnkey managed" you mean, you """just""" """configure""" """IP addresses""" and it """all just works"""? You can certainly install Ubuntu on a very powerful machine with a WAN interface (e.g., a NIC connected to a residential cable internet connection). Then, us…
> You don't need a "LoadBalancer", you can specify that nginx-controller runs on the host network of specifically the machine with the WAN interface and configure its service's external IP to the WAN IP, and now you support Ingress.
And if you need to service that machine or it goes down?
> But how do you imagine having multiple LoadBalancer resources without multiple IPs? And how do you imagine having multiple IPs without ARIN?
Most colos/transit providers will happily lend you their IPs
Re: Oxide Computer: Docs
#115As an on-prem sysadmin, I'm not sure I will ever in my life work on an environment that has the minimum specifications of an Oxide system. Every entire datacenter I've worked in has less total capacity than a single "sled" here.
Are there intentions to go smaller than this?
Re: Oxide Computer: Docs
#116Earlier quoted context omitted.
> I imagine it must be a bit like what working with minicomputers in 70s thru the 90s was like. I had a chance to talk to someone who worked at Oxide once. I got excited because my experience matches up nicely with their products on several levels. The person got excited as well and started talking about how to interview there, but the conversation got kind of weird. They kept emphasizing how important it was to not…
This conversation strikes me as unlikely on several levels. First, no one would have coached you on "how to interview at Oxide" because that's not where the process starts -- it starts with you preparing your materials.[0] (Our review of the materials constitutes ~95% of our process.) Second, we have always been very explicit about compensation (that is, we ourselves brought it up early in conversations); no one at O…
Hey Bryan, I enjoyed reading the cash compensation article, but I'm curious about how it meshes with your equity compensation? Is it awarded purely on when people joined? Does everyone who wasn't a founder get the same amount? It feels like you could run into much of the same problems the article points out about transparency and so on if the equity component isn't as straight-forward?
Re: Oxide Computer: Docs
#117Earlier quoted context omitted.
One benefit is that they're competing in an industry where the time to get a sled up and running for compute can be measured in weeks (they say they've heard up to 90 days) whereas their solution is basically plug and play - Bryan was trying to get Steve to admit set up took "hours" whereas Steve was hedging and saying customers could get started "within a week." They go into more details on their podcast, and this s…
Do I have it right that they ship their own hypervisor (that's based on maybe Solaris, not KVM) as firmware? Let's assume a very small team can compete on a technical level, it still seems like it could cut out a lot of the potential market. I can't imagine that large cloud / "web scale" companies would want that. Most want a fair bit of control of their own hypervisor and management stacks based around KVM. And "ent…
I think quite often when we assume 'most want to fair bit of control' is just not true. Enterprises want something that just works, they want control if they can't have something that just works.
If you have a team that is struggling building an internal cloud with all this control (and problems) and all this commodity hardware (and its problems) then maybe they would be happy to switch to something that just works.
> And "enterprise" type companies are going to have issues with certification I would have thought -- will RedHat, Microsoft, SAP, Oracle, etc certify their supported products on top of this hypervisor? Seems like a difficult and expensive process.
If that was the case and nobody running any of these would run their rack, then I wouldn't think they would not have received any funding. But I don't know enough about these certification process to really comment.
> Companies that support their own virtual machine software but don't support their own hypervisor and don't like what's available from vmware or Microsoft or RedHat
Non of these come with a fully integrated rack.
The competition would be somebody willing to buy a rack of Dell servers with VMWare software. Or somebody willing to buy a rack of Dell server and then use RedHat and set up all their own cloud style infrastructure.
Re: Oxide Computer: Docs
#118Earlier quoted context omitted.
[flagged]
We didn't shadowban them, we told them we were banning them: https://news.ycombinator.com/item?id=34916347 , we told them why, and it wasn't the reason you've presumed. When people presume things about moderation, they almost always get it wrong. That's a pity because all anyone has to do to get the correct answer is ask us. If you see an account that's banned and you don't think it should be, please let us know at h…
[dead]
Re: Oxide Computer: Docs
#119I am super interested in learning more about the storage subsystem! I figured they'd be using ZFS, given the people involved, but it appears they've also gone ahead and built a clustered FS (crucible) on top of it? I figured something like that would be necessary to handle fault tolerance at the gimlet level. (Losing an entire shelf / drive controller, etc.) Getting ZFS to go multi-node is surely a neat trick. Second…
They use Crucible on top of ZFS. https://github.com/oxidecomputer/crucible
I don't think they have anything for S3-like service but there are other options for that, e.g. https://garagehq.deuxfleurs.fr or MinIO.
I am not sure whether they have their own SSDs or use of the shelf SSDs just with their firmware or something.
Re: Oxide Computer: Docs
#120Earlier quoted context omitted.
One benefit is that they're competing in an industry where the time to get a sled up and running for compute can be measured in weeks (they say they've heard up to 90 days) whereas their solution is basically plug and play - Bryan was trying to get Steve to admit set up took "hours" whereas Steve was hedging and saying customers could get started "within a week." They go into more details on their podcast, and this s…
Do I have it right that they ship their own hypervisor (that's based on maybe Solaris, not KVM) as firmware? Let's assume a very small team can compete on a technical level, it still seems like it could cut out a lot of the potential market. I can't imagine that large cloud / "web scale" companies would want that. Most want a fair bit of control of their own hypervisor and management stacks based around KVM. And "ent…
The hypervisor is based on Bhyve from FreeBSD + Propolis in user space. Illumos actually has/ had KVM and there is a talk by Bryan Cantrill where he speaks about the porting effort. All of that information is readily searchable.