Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes on Oxide: How customer needs shaped our integrations

oxide.computer

Post reply on HN
Print view View original

31–40 of 70 posts

Re: Kubernetes on Oxide: How customer needs shaped our integrations

#31

Post by sudomateo »

bitlad wrote: post #19

I am wondering when would you use kubernetes on oxide vs running kubernetes with kubevirt on baremetal? At first look, it feels like oxide is equivalent to proxmox or some virtualization tool, may be it uses qemu stuff underneath. Just curious. The reason I am asking this is that, we have lot onprem scenarios in our business. We are tightly coupled with k8s, to solve this we started building an internal project that…

Good question. We use our own hypervisor[0] that's not KVM/QEMU. We don't have nested virtualization today so we don't follow the KubeVirt model, though we are discussing what CRDs like OxideInstance would look like for those that want to operate solely in Kubernetes manifests.

The core primitive on Oxide is the instance (virtual machine). We could support some container primitive, but that's a larger product direction discussion. Our host OS is Helios (Illumos) so there are details to iron out there regarding what abstractions we would build and expose to the users. Not impossible but not something we're currently pursuing either given that we have other immediate product asks.

If you're at a scale where compute density, power efficiency, security, and rack-level API management matters then that's where Oxide makes sense for you.

[0] https://github.com/oxidecomputer/propolis

Re: Kubernetes on Oxide: How customer needs shaped our integrations

#32

Post by bitlad »

bakies wrote: post #25
bitlad wrote: post #19

I am wondering when would you use kubernetes on oxide vs running kubernetes with kubevirt on baremetal? At first look, it feels like oxide is equivalent to proxmox or some virtualization tool, may be it uses qemu stuff underneath. Just curious. The reason I am asking this is that, we have lot onprem scenarios in our business. We are tightly coupled with k8s, to solve this we started building an internal project that…

Yeah me too. Like I kinda compare oxide to coreweave and coreweave is built on top of k8s afaik. I'm wondering what the underlying tech oxide leans on. If I was doing this I would be using talos and k8s on baremetal and building on top of that. Container first rather than VM first seems like a big advantage imo. Most workloads will be containers.

Coreweave is a cloud provider right? I dont think it is apple to apple comparison.

Re: Kubernetes on Oxide: How customer needs shaped our integrations

#33

Post by dcre »

mixmastamyk wrote: post #30
bcantrill wrote: post #5

Earlier quoted context omitted.

Are you referring to our RFD system?[0][1] Because if so, good news![2] [0] https://rfd.shared.oxide.computer/rfd/0001 [1] https://oxide-and-friends.transistor.fm/episodes/rfds-the-ba... [2] https://github.com/oxidecomputer/rfd-site

Why would a “doc system” use React?

Because it’s a web site.

Re: Kubernetes on Oxide: How customer needs shaped our integrations

#34

Post by donatj »

Bug report I guess, but the pages top navigation seems completely inoperable to me on my iPad. Clicking with my finger does not work. Clicking with my Apple Pencil does not work. Any sort of hover state that might be there does not work (with the pencil)

Re: Kubernetes on Oxide: How customer needs shaped our integrations

#35

Post by dilyevsky »

sudomateo wrote: post #29
redwood wrote: post #20

How are folks managing stateful workloads like databases on Oxide?

I cover this a bit in the post. Today they are running something like Longhorn backed by Oxide storage. When we release our CSI plugin they can use that instead.

why longhorn and not mayastor? longhorn doesn't even have nvmeof that is not experimental

Re: Kubernetes on Oxide: How customer needs shaped our integrations

#36

Post by lkasjdas »

pianoben wrote: post #4

I have seldom wanted anything as much as I want an Oxide rack at home. Maybe in 40 years we'll start to see the first ones show up in surplus auctions...

Every single Oxide article has comments about people wanting one at home.

A great example of completely misdirected marketing and/or engineering. Oxide should have developed home microcomputers, and they would have sold like cupcakes, with their ASCII art marketing. Instead they do million dollar "mainframes" that no one (except VCs) wants to buy.

Re: Kubernetes on Oxide: How customer needs shaped our integrations

#37

Post by quadrifoliate »

lkasjdas wrote: post #36
pianoben wrote: post #4

I have seldom wanted anything as much as I want an Oxide rack at home. Maybe in 40 years we'll start to see the first ones show up in surplus auctions...

Every single Oxide article has comments about people wanting one at home. A great example of completely misdirected marketing and/or engineering. Oxide should have developed home microcomputers, and they would have sold like cupcakes, with their ASCII art marketing. Instead they do million dollar "mainframes" that no one (except VCs) wants to buy.

I'm sure they have done plenty of market research, and that research probably indicates that enthusiasts who buy server racks are a much smaller market than large enterprise users that need easily scalable on-prem resources.

Lots of people saying "I would love this" is not a profitable market. There are a lot of implicit assumptions in that phrase. Would you buy the lowest level rack for $100k, paying extra for any support? (I don't have any insight into actual pricing, but I know enterprise consumers don't bat an eyelid at prices like that.)

If not then maybe the "I would love this" is not relevant for a real market.

Re: Kubernetes on Oxide: How customer needs shaped our integrations

#38

Post by bcantrill »

lkasjdas wrote: post #36
pianoben wrote: post #4

I have seldom wanted anything as much as I want an Oxide rack at home. Maybe in 40 years we'll start to see the first ones show up in surplus auctions...

Every single Oxide article has comments about people wanting one at home. A great example of completely misdirected marketing and/or engineering. Oxide should have developed home microcomputers, and they would have sold like cupcakes, with their ASCII art marketing. Instead they do million dollar "mainframes" that no one (except VCs) wants to buy.

(I think you meant "like hotcakes", not "like cupcakes"?)

In any event: from the beginning, we have always been targeted at the enterprise buyer who is looking at annual public cloud bills in the tens, hundreds or (in some cases) thousands of millions of dollars per year. We love the enthusiasm that Oxide engenders among the home lab set, but that's not how we have geared the business (for lots of good reasons).

Also, for whatever it's worth: people do want to buy them, it turns out.

Re: Kubernetes on Oxide: How customer needs shaped our integrations

#39

Post by esseph »

sudomateo wrote: post #12
thegagne wrote: post #7

Earlier quoted context omitted.

Yes this would be great addition! Also it would be awesome to see proper load balancer, ingress controller with `gateway api` and a `api gateway` (two separate things). I’m not an oxide customer, just a fan. I have lots of ideas around how something like this should look, being disappointed with the complexity and also shortcomings of products on the market for this stuff, both in the cloud and on prem.

I'll need to record a video on what's possible today. My personal Kubernetes cluster running on Oxide uses the CCM for LoadBalancer services and NGINX Gateway Fabric for Gateway API things. I'm mostly using HTTPRoute resources today. There's an opportunity to more tightly integrate at the network layer but we'll want to get our load balancer released first.

> My personal Kubernetes cluster running on Oxide

What a flex.

It's gotten to the point I'm starting to actively look for Oxide customers so I can apply with them.

Re: Kubernetes on Oxide: How customer needs shaped our integrations

#40

Post by esseph »

wmf wrote: post #26
bitlad wrote: post #19

I am wondering when would you use kubernetes on oxide vs running kubernetes with kubevirt on baremetal? At first look, it feels like oxide is equivalent to proxmox or some virtualization tool, may be it uses qemu stuff underneath. Just curious. The reason I am asking this is that, we have lot onprem scenarios in our business. We are tightly coupled with k8s, to solve this we started building an internal project that…

You would use Oxide if you hate Dell/HPE/Lenovo/Supermicro. Also most people won't run k8s on bare metal because they want to dynamically provision a bunch of clusters.

You treat the hardware as a software problem and provision it dynamically. Also keeps I/O high if you have data locality (HCI).
Post reply on HN