Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We're Leaving Kubernetes

gitpod.io

Post reply on HN
Print view View original

31–40 of 348 posts

Re: We're Leaving Kubernetes

#31

Post by loujaybee »

sofixa wrote: post #22
xyst wrote: post #15

I do agree with the points in article that k8s is not a good fit for development environments. In my opinion, k8s is great for stable and consistent deployment/orchestration of applications. Dev environments by default are in a constant state of flux. I don’t understand the need for “cloud development environments” though. Isn’t the point of containerized apps is to avoid the need for synchronizing dev envs amongst t…

It's to ensure a consistent environment for all developers, with the resources required. E.g. they mention GPUs, for developers working with GPU-intensive workloads. You can ship all developers gaming laptops with 64GB RAM and proper GPUs, and have them fight the environment to get the correct libraries as you have in prod (even with containers that's not trivial), or you can ship them Macbook Airs and similar, and h…

Yeah, exactly. Containers locally are a basic foundation. But usually those containers or services need to talk to one another, they need some form of auth and credentials, they need some networking setup. There's a lot of configuration in all of that. The more devs swap projects or the more complex the thing you're working on the more the challenge grows. Automating depedencies, secret access, ensuring projects have the right memory, cpu, gpu etc. Also security - moving source code off your laptop and devices and standardizing your setups helps if you need to do a lot of audit and compliance as you can automate it.

Re: We're Leaving Kubernetes

#32

Post by roshbhatia »

xyst wrote: post #15

I do agree with the points in article that k8s is not a good fit for development environments. In my opinion, k8s is great for stable and consistent deployment/orchestration of applications. Dev environments by default are in a constant state of flux. I don’t understand the need for “cloud development environments” though. Isn’t the point of containerized apps is to avoid the need for synchronizing dev envs amongst t…

In my experience, the case where this becomes really valuable is if your team needs access to either different kinds of hardware or really expensive hardware that changes relatively quickly (i.e. GPUs). At a previous small startup I setup https://devpod.sh/ (similar to gitpod) for our MLE/Data team. It was a big pro to leverage our existing k8s setup w/ little configuration needed to get these developer envs up and running as-needed, and we could piggyback off of our existing cost tracking tooling to measure usage, but I do feel like we already had infra conducive to running dev envs on k8s before making this decision -- we had cost tracking tooling, we had a dedicated k8s cluster for tooling, we had already been supporting GPU based workloads in k8s, and our platform team that managed all the k8s infra also were the SMEs for anything devenv releated. In a world where we started fresh and absolutely needed ephemeral devenvs, I think the native devcontainer functionality in vscode or something like github codespaces would have been our go to, but even then I'd push for a docker-compose based workflow prior to touching any of these other tools.

The rest of our eng team just did dev on their laptops though. I do think there was a level of batteries-included-ness that came with the ephemeral dev envs which our less technical data scientists appreciated, but the rest of our developers did not. Just my 2c

Re: We're Leaving Kubernetes

#33

Post by horsawlarway »

Personally - just let the developer own the machine they use for development.

If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine.

I have seen several attempts to move dev environments to a remote host. They invariably suck.

Yes - that means you need to pay for decent hardware for your devs, it's usually cheaper than remote resources (for a lot of reasons).

Yes - that means you need to support running your stack locally. This is a good constraint (and a place where containers are your friend for consistency).

Yes - that means you need data generation tooling to populate a local env. This can be automated relatively well, and it's something you need with a remote env anyways.

---

The only real downside is data control (ie - the company has less control over how a developer manages assets like source code). I'm my experience, the vast majority of companies should worry less about this - your value as a company isn't your source code in 99.5% of cases, it's the team that executes that source code in production.

If you're in the 0.5% of other cases... you know it and you should be in an air-gapped closed room anyways (and I've worked in those too...)

Re: We're Leaving Kubernetes

#34

Post by javier_e06 »

The article is an excellent cautionary tale. Debugging an app in a container is one thing. Debugging and app running inside a Kubernetes node is a rabbit hole that demands more hours and expertise.

Re: We're Leaving Kubernetes

#35

Post by dangsux »

horsawlarway wrote: post #33

Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent h…

[dead]

Re: We're Leaving Kubernetes

#36

Post by rglover »

bhouston wrote: post #11
rglover wrote: post #10

Earlier quoted context omitted.

What stack are you deploying?

Stuff like this, just at larger scale: https://github.com/bhouston/template-typescript-monorepo This is my living template of best practices.

I'd investigate getting a build out to Node.js (looks like you already have this) and then just doing a simple SCP of the build to a VPS. From there, just use a systemd script to handle startup/restart on errors. For logging, something like the Winston package does the trick.

If you want some guidance, shoot me an email (in profile). You can run most stuff for peanuts.

Re: We're Leaving Kubernetes

#37

Post by shriek »

horsawlarway wrote: post #33

Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent h…

And the reason they suck is the feedback loop is just too high as compared to running it locally. You have to jump through hoops to debug/troubleshoot your code or any issues that you come across between your code and output of your code. And it's almost impossible to work on things when you have spotty internet. I haven't worked on extremely sensitive data but for PII data from prod to dev, scrubbing is a good practice to follow. This will vary based on the project/team you're on of course.

Re: We're Leaving Kubernetes

#38

Post by binary132 »

horsawlarway wrote: post #33

Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent h…

Hello. Currently debugging my kubernetes-based dev pod and not getting anything else done. What fun!

Re: We're Leaving Kubernetes

#39

Post by Imustaskforhelp »

bhouston wrote: post #6

I also recently left Kubernetes. It was a huge waste of time and money. I've replaced it with just a series of services on Google Cloud Run and then using Google's Cloud Run Tasks services for longer running tasks. The infrastructure now incredibly understandable and simple and cost effective. Kubernetes cost us >$million in both DevOps time and actually Google Cloud costs unnecessarily, and even worse it cost us tim…

wow very very interesting. I think we can discuss about it on hours.

1.) What would you think of things like hetzner / linode / digitalocean (if stable work exists)

2.) What do you think of https://sst.dev/ or https://encore.dev/ ? (They support rather easier migration)

3.) Could you please indicate the split of that 1 million $ in devops time and google cloud costs unnecessarily & were there some outliers (like oh our intern didn't add this specific variable and this misconfigured cloud and wasted 10k on gcloud oops! or was it , that bandwidth causes this much more in gcloud (I don't think latter to be the case though))

Looking forward to chatting with you!

Re: We're Leaving Kubernetes

#40

Post by ethbr1 »

kitd wrote: post #4
lolinder wrote: post #3

> This is not a story of whether or not to use Kubernetes for production workloads that’s a whole separate conversation. As is the topic of how to build a comprehensive soup-to-nuts developer experience for shipping applications on Kubernetes. > This is the story of how (not) to build development environments in the cloud. I'd like to request that the comment thread not turn into a bunch of generic k8s complaints. Th…

Agreed. It's actually a very interesting use case and I can easily see that K8s wouldn't be the answer. My dev env is very definitely my "pet", thank you very much!

It'd be nice to editorialize the title a bit with "... (for dev envs)" for clarity.

Super useful negative example, and the lengths they pursued to make it fit! And no knock on the initial choice or impressive engineering, as many of the k8s problems they hit likely weren't understood gaps at the time they chose k8s.

Which makes sense, given k8s roots in (a) not being a security isolation tool & (b) targeting up-front configurability over runtime flexibility.

Neither of which mesh well with the co-hosted dev environment use case.

Post reply on HN