Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We're Leaving Kubernetes

gitpod.io

Post reply on HN
Print view View original

41–50 of 348 posts

Re: We're Leaving Kubernetes

#41

Post by csweichel »

lmeyerov wrote: post #25

I was intrigued because the development environment problem is similar to the data scientist one - data gravity, GPU sharing, etc - but I'm confused on the solution? Oddly, I left with a funny alternate takeaway: One by one, their clever inhouse tweaks & scheduling preferences were recognized by the community and turned into standard k8s knobs So I'm back to the original question... What is fundamentally left? It sou…

OP here. The Kubernetes community has been fantastic at evolving the platform, and we've greatly enjoyed being in the middle of it. Indeed, many of the things we had to build next to Kubernetes have now become part of k8s itself.

Still, some of the core challenges remain: - the flexibility Kubernetes affords makes it hard to build and distribute a product with such specific requirements across the broad swath of differently set up Kubernetes installations. Managed Kubernetes services help, but come with their own restrictions (e.g. Kernel versions on GKE). - state handling and storage remains unsolved. PVCs are not reliable enough, subject to a lot of variance (see point above), and depending on the backing storage have vastly different behaviour. Local disks (which we use to this day), make workspace startup and backup expensive from a resource perspective and hard to predict timing wise. - user namespaces have come a long way in Kubernetes, but by themselves are not enough. /proc is still masked, FUSE is still not usable. - startup times, specifically container pulls and backup restoration, are hard to optimize because they depend on a lot of factors outside of our control (image homogeneity, cluster configuration)

Fundamentally, Kubernetes simply isn't the right choice here. It's possible to make it work, but at some point the ROI of running on Kubernetes simply isn't there.

Re: We're Leaving Kubernetes

#42

Post by ethbr1 »

shriek wrote: post #37
horsawlarway wrote: post #33

Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent h…

And the reason they suck is the feedback loop is just too high as compared to running it locally. You have to jump through hoops to debug/troubleshoot your code or any issues that you come across between your code and output of your code. And it's almost impossible to work on things when you have spotty internet. I haven't worked on extremely sensitive data but for PII data from prod to dev, scrubbing is a good pract…

Aka 'if a developer knew beforehand everything they needed, it wouldn't be development'

Re: We're Leaving Kubernetes

#43

Post by Imustaskforhelp »

rglover wrote: post #36
bhouston wrote: post #11

Earlier quoted context omitted.

Stuff like this, just at larger scale: https://github.com/bhouston/template-typescript-monorepo This is my living template of best practices.

I'd investigate getting a build out to Node.js (looks like you already have this) and then just doing a simple SCP of the build to a VPS. From there, just use a systemd script to handle startup/restart on errors. For logging, something like the Winston package does the trick. If you want some guidance, shoot me an email (in profile). You can run most stuff for peanuts.

yeh I have same thoughts , also if possible , bun can also reduce memory usage in very very basic scenarios https://www.youtube.com/watch?v=yJmyYosyDDM

Or just https://github.com/mightymoud/sidekick or coolify or dokku or dockify , like there are million of such things , oh just remembered kamala deploy from DHH and docker swarm IIRC (though people have seemed to forget docker swarm !)

I like this idea very much !

Re: We're Leaving Kubernetes

#44

Post by Imustaskforhelp »

rglover wrote: post #36
bhouston wrote: post #11

Earlier quoted context omitted.

Stuff like this, just at larger scale: https://github.com/bhouston/template-typescript-monorepo This is my living template of best practices.

I'd investigate getting a build out to Node.js (looks like you already have this) and then just doing a simple SCP of the build to a VPS. From there, just use a systemd script to handle startup/restart on errors. For logging, something like the Winston package does the trick. If you want some guidance, shoot me an email (in profile). You can run most stuff for peanuts.

there was some recent HN post which showed that they didn't even use docker but rather there was some other mechanism and it was so so simple , I really enjoyed that article

Re: We're Leaving Kubernetes

#45

Post by ethbr1 »

nickstinemates wrote: post #28
loujaybee wrote: post #23

Earlier quoted context omitted.

Yeah, that's fair. The blog was getting quite long, so we need to do some deeper dives in follow-ups. Gitpod Flex is runner-based. The runner interface is intentionally generic so that we can support different clouds, on-prem or just Linux in future. The first implemented runner is built around AWS primitives like EC2, EBS and ECS. But because of the more generic interface Gitpod now supports local / desktop environm…

Echoing the parent you're replying to. You built up all of the context and missed they payoff.

I thought it was fair.

>> We’ll be posting a lot more about Gitpod Flex architecture in the coming weeks or months.

Cramming more detail into this post would have exceeded the average user read time ceiling.

Re: We're Leaving Kubernetes

#46

Post by pphysch »

The problem with "development environments", like other interactive workloads, is that there is a human at the other end that desires a good interactive experience with every keypress. It's a radically different problem space than what k8s was designed for.

From a resource provider productive, the only way to squeeze a margin out of that space would be to reverse engineer 100% of human developer behavior so that you can ~perfectly predict "slack" in the system that could be reallocated to other users. Otherwise it's just a worse DX, like TFA gives examples of. Not a business I'm envious too be in... Just give everyone a dedicated VM or desktop, and make sure there's a batch system for big workloads.

Re: We're Leaving Kubernetes

#47

Post by bhouston »

rglover wrote: post #36
bhouston wrote: post #11

Earlier quoted context omitted.

Stuff like this, just at larger scale: https://github.com/bhouston/template-typescript-monorepo This is my living template of best practices.

I'd investigate getting a build out to Node.js (looks like you already have this) and then just doing a simple SCP of the build to a VPS. From there, just use a systemd script to handle startup/restart on errors. For logging, something like the Winston package does the trick. If you want some guidance, shoot me an email (in profile). You can run most stuff for peanuts.

> I'd investigate getting a build out to Node.js (looks like you already have this) and then just doing a simple SCP of the build to a VPS. From there, just use a systemd script to handle startup/restart on errors. For logging, something like the Winston package does the trick. If you want some guidance, shoot me an email (in profile). You can run most stuff for peanuts.

I appreciate the offer! But it is not as robust and it is more expensive and misses a lot of benefits.

Back in the 1990s I did FTP my website to a VPS after I graduated from Geocities.

Google Cloud charges based on CPU used. Thus my servers have no traffic, they cost less than a $1/month. If they have traffic, they are still cost effective. https://web3dsurvey.com has about 500,000 hits per month and it costs me $4/month to run both the Remix web server and the Fastify API server. Details here: https://x.com/benhouston3d/status/1840811854911668641

Also it will autoscale under load. Thus when one of my posts was briefly the top story on Hacker News last month, Google Cloud Run added more instances to my server to handle the load (because I do not run my personal site behind a CDN, it cost too much, I prefer to pay $1/month for hosting.)

Also deploying Docker containers that build on Github Actions CI in a few minutes is a great automated experience.

I do also use Google services like Cloud Storage, Firestore, BigQuery etc. And it is easier to just run it on GCP infrastructure for speed.

I also have to version various tools that get installed in the docker like Blender, Chromium, etc. This is the perfect use case for Docker.

I feel this is pretty close to optimal. Fast, cheap, scalable, automated and robust.

Re: We're Leaving Kubernetes

#48

Post by [deleted] »

horsawlarway wrote: post #33

Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent h…

[deleted]

Re: We're Leaving Kubernetes

#49

Post by haolez »

horsawlarway wrote: post #33

Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent h…

Sometimes I don't even use virtual envs when developing locally in Python. I just install everything that I need with pip --user and be done with it. Never had any conflicts with system packages whatsoever. If I somehow break my --user environment, I simply delete it and start again. Never had any major version mismatch in dependencies between my machine and what was running in production. At least not anything that would impact the actual task that I was working on.

I'm not recommending this as a best practice. I just believe that we, as developers, end up creating some myths to ourselves of what works and what doesn't. It's good to re-evaluate these beliefs now and then.

Re: We're Leaving Kubernetes

#50

Post by csweichel »

horsawlarway wrote: post #33

Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent h…

OP here. There definitely is a place for running things on your local machine. Exactly as you say: one can get a great deal of consistency using VMs.

One of the benefits of moving away from Kubernetes, to a runner-based architecture , is that we can now seamlessly support cloud-based and local environments (https://www.gitpod.io/blog/introducing-gitpod-desktop).

What's really nice about this is that with this kind of integration there's very little difference in setting up a dev env in the cloud or locally. The behaviour and qualities of those environments can differ vastly though (network bandwidth, latency, GPU, RAM, CPUs, ARM/x86).

Post reply on HN