Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We're Leaving Kubernetes

gitpod.io

Post reply on HN
Print view View original

51–60 of 348 posts

Re: We're Leaving Kubernetes

#51

Post by lotharcable »

horsawlarway wrote: post #33

Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent h…

I strongly recommend just switching the Dev environment over to Linux and taking advantage of tools like "distrobox" and "toolbx".

https://github.com/89luca89/distrobox

https://containertoolbx.org/

It is sorta like Vagrant, but instead of using virtualbox virtual machines you use podman containers. This way you get to use OCI images for your "dev environment" that integrates directly into your desktop.

https://podman.io/

There is some challenges related to usermode networking for non-root-managed controllers and desktop integration has some additional complications. But besides that it has almost no overhead and you can have unfettered access to things like GPUs.

Also it is usually pretty easy to convert your normal docker or kubernetes containers over to something you can run on your desktop.

Also it is possible to use things like Kubernetes pods definitions to deploy sets of containers with podman and manage it with systemd and such things. So you can have "clouds of containers" that your dev container needs access to locally.

If there is a corporate need for window-specific applications then running Windows VMs or doing remote applications over RDP is a possible work around.

If everything you are targeting as a deployment is going to be Linux-everything then it doesn't make a lot of sense to jump through a bunch of hoops and cause a bunch of headaches just to avoid having it as workstation OS.

Re: We're Leaving Kubernetes

#52

Post by idunnoman1222 »

horsawlarway wrote: post #33

Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent h…

Sounds like you are not using a lot of hardware - Rfid, POS, top-spec video cards, etc

Re: We're Leaving Kubernetes

#53

Post by lmeyerov »

csweichel wrote: post #41
lmeyerov wrote: post #25

I was intrigued because the development environment problem is similar to the data scientist one - data gravity, GPU sharing, etc - but I'm confused on the solution? Oddly, I left with a funny alternate takeaway: One by one, their clever inhouse tweaks & scheduling preferences were recognized by the community and turned into standard k8s knobs So I'm back to the original question... What is fundamentally left? It sou…

OP here. The Kubernetes community has been fantastic at evolving the platform, and we've greatly enjoyed being in the middle of it. Indeed, many of the things we had to build next to Kubernetes have now become part of k8s itself. Still, some of the core challenges remain: - the flexibility Kubernetes affords makes it hard to build and distribute a product with such specific requirements across the broad swath of diff…

Thanks!

AFAICT, a lot of that comes down to storage abstractions, which I'll be curious to see the answer on! Pinned localstorage cloud native is frustrating.

I sense another big chunk is the fast secure start problems that firecracker (noted in the blogpost) solve but k8s is not currently equipped for. Our team has been puzzling that one for awhile, and part of our guess is incentives. It's been 5+ years since firecracker came out, so likewise been frustrating to see.

Re: We're Leaving Kubernetes

#55

Post by geoctl »

I've worked on something similar to gitpod in a slightly different context that's part of a much bigger personal project related to secure remote access that I've actually spent a few years building now and hope to open source in a few months from now. While I agree on many of the points in the article, I just don't understand how using micro VMs by itself replaces K8s unless they actually start building their own K8s that orchestrates their micro VMs (as opposed to containers in the case of k8s) ending up with the same thing basically when k8s itself can be used to orchestrate the outer containers that run the micro VMs used to run the dev containers. Yes, k8s has many challenges when it comes to nesting containers, cgroups, creating rootless containers inside the outer k8s containers and other stuff such as multi-region scaling, but actually the biggest challenge that I've faced so far isn't related to networkPolicies or cgroups but is actually by far related to storage, both when it comes to (lazily) pulling big OCI images which are extremely unready to be used for dev containers whose sizes are typically in the GBs or 10s of GBs as well as also when it comes to storage virtualization over the underlying k8s node storage. There are serious attempts to accelerate image pulling (e.g. Nydus) but such solutions would still probably be needed whether you use micro VMs or rootless/userns containers in order to load and run your dev containers.

Re: We're Leaving Kubernetes

#56

Post by edplmajor »

csweichel wrote: post #50
horsawlarway wrote: post #33

Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent h…

OP here. There definitely is a place for running things on your local machine. Exactly as you say: one can get a great deal of consistency using VMs. One of the benefits of moving away from Kubernetes, to a runner-based architecture , is that we can now seamlessly support cloud-based and local environments ( https://www.gitpod.io/blog/introducing-gitpod-desktop ). What's really nice about this is that with this kind…

Need middleware libs that react to eBPF data and signal app code to scale up/down forks in their own memory VM, like V8

Kubernetes is another mess of userspace ops tools. Userspace is for composable UI not backend. Kube and Chef and all those other ops tools are backend functionality being used like UI by leet haxxors

Re: We're Leaving Kubernetes

#57

Post by 0xbadcafebee »

horsawlarway wrote: post #33

Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent h…

Most teams/products I have been involved in, the stack always grows to the point that a dev can no longer test it on their own machine, regardless of how big the machine is. And having a different development machine than production leads to completely predictable and unavoidable problems. Devs need to create the software tooling to make remote dev less painful. I mean, they're devs... making software is kind of their whole thing.

Re: We're Leaving Kubernetes

#59

Post by andreweggleston »

rekoros wrote: post #54

We've been using Nix flakes and direnv ( https://direnv.net/ ) for developer environments and NixOS with https://github.com/serokell/deploy-rs for prod/deploys - takes serious digging and time to set up, but excellent experience with it so far.

I’ve been using Nix for the past year and it really feels like the holy grail for stable development environments. Like you said—it takes serious time to set up, but it seems like that’s an unavoidable reality of easily sharable dev envs.
Post reply on HN