Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We're Leaving Kubernetes

gitpod.io

Post reply on HN
Print view View original

21–30 of 348 posts

Re: We're Leaving Kubernetes

#21

Post by spwa4 »

bhouston wrote: post #9
elcomet wrote: post #7

Earlier quoted context omitted.

Aren't you afraid of being now stuck with GCP?

It is just a bunch of docker containers. Some run in tasks and some run as auto-scaling services. Would probably take a week to switch to AWS as there are equivalent managed services there. But this is really a spurious concern. I myself used to care about it years ago. But in practice, rarely do people switch between cloud providers because the incremental benefits are minor, they are nearly equivalent, there is not…

How does the orchestration work? How do you share storage? How do the docker containers know how to find each other? How does security work?

I feel like Kubernetes' downfall, for me, is the number of "enterprise" features it (got convinced into) supporting and enterprise features doing what they do best: turning the simplest of operations into a disaster.

Re: We're Leaving Kubernetes

#22

Post by sofixa »

xyst wrote: post #15

I do agree with the points in article that k8s is not a good fit for development environments. In my opinion, k8s is great for stable and consistent deployment/orchestration of applications. Dev environments by default are in a constant state of flux. I don’t understand the need for “cloud development environments” though. Isn’t the point of containerized apps is to avoid the need for synchronizing dev envs amongst t…

It's to ensure a consistent environment for all developers, with the resources required. E.g. they mention GPUs, for developers working with GPU-intensive workloads. You can ship all developers gaming laptops with 64GB RAM and proper GPUs, and have them fight the environment to get the correct libraries as you have in prod (even with containers that's not trivial), or you can ship them Macbook Airs and similar, and have them run consistent (the same) dev environments remotely (you can self-host gitpod, it's not only a cloud service, it's more the API/environment to get consistent remote dev enviornments).

Re: We're Leaving Kubernetes

#23

Post by loujaybee »

ensignavenger wrote: post #8

The article does a great job of explaining the challenges they ran into with Kubernetes, and some of the things they tried... but I feel like it drops the ball at the end by not telling us at least a little what they chose instead. The article mentions they call their new solution "Gitpod Flex" but there is nothing about what Gitpod Flex is. They said they tried microVMs and decided against them, and of course Kubern…

Yeah, that's fair. The blog was getting quite long, so we need to do some deeper dives in follow-ups.

Gitpod Flex is runner-based. The runner interface is intentionally generic so that we can support different clouds, on-prem or just Linux in future.

The first implemented runner is built around AWS primitives like EC2, EBS and ECS. But because of the more generic interface Gitpod now supports local / desktop environments on MacOS. And again, future OS support will come.

There’s a bit more information in the docs, but we will do some follow ups!

- https://www.gitpod.io/docs/flex/runners/aws/setup-aws-runner... - https://www.gitpod.io/docs/flex/gitpod-desktop

(I work at Gitpod)

Re: We're Leaving Kubernetes

#24

Post by bhouston »

spwa4 wrote: post #21
bhouston wrote: post #9

Earlier quoted context omitted.

It is just a bunch of docker containers. Some run in tasks and some run as auto-scaling services. Would probably take a week to switch to AWS as there are equivalent managed services there. But this is really a spurious concern. I myself used to care about it years ago. But in practice, rarely do people switch between cloud providers because the incremental benefits are minor, they are nearly equivalent, there is not…

How does the orchestration work? How do you share storage? How do the docker containers know how to find each other? How does security work? I feel like Kubernetes' downfall, for me, is the number of "enterprise" features it (got convinced into) supporting and enterprise features doing what they do best: turning the simplest of operations into a disaster.

> How does the orchestration work?

Github Actions CI. Take this and make a few more dependencies and a matrix strategy and you are good to go: https://github.com/bhouston/template-typescript-monorepo/blo... For dev environments, you can add post-fixes to the services based on branches.

> How do you share storage?

I use managed DBs and Cloud Storage for shared storage. I think that provisioning your own SSDs/HDs to the cloud is indicative of an anti-pattern in your architecture.

> How do the docker containers know how to find each other?

I try to avoid too much communication between services directly, rather try to go through pub-sub or similar. But you can set up each service with a domain name and access them that way. With https://web3dsurvey.com, I have an api on https://api.web3dsurvey.com and then a review environment (connected to the main branch) with https://preview.web3dsurvey.com / https://api.preview.web3dsurvey.com.

> How does security work?

You can configure Cloud Run services to be internal only and not to accept outside connections. Otherwise one can just use JWT or whatever is normal on your routes in your web server.

Re: We're Leaving Kubernetes

#25

Post by lmeyerov »

I was intrigued because the development environment problem is similar to the data scientist one - data gravity, GPU sharing, etc - but I'm confused on the solution?

Oddly, I left with a funny alternate takeaway: One by one, their clever inhouse tweaks & scheduling preferences were recognized by the community and turned into standard k8s knobs

So I'm back to the original question... What is fundamentally left? It sounds like one part is maintaining a clean container path to simplify a local deploy, which a lot of k8s teams do (ex: most of our enterprise customers prefer our docker compose & AMIs over k8s). But more importantly, something fundamental architecturally about how envs run that k8s cannot do, but they do not identify?

Re: We're Leaving Kubernetes

#26

Post by debarshri »

Phew, it is absolutely true. Building dev environments on k8s become wasteful. To add to this complexity, if you are building a product that is self hosted on customer's infrastructure. Debugging and support also become non homogeneous and difficult.

What we have seen works especially when you are building developer centric product is expose these native issues around network, memory, compute and storage to engineers and they are more willing to work around it. Abstracting those issues leads to shift in responsibility on the product.

Having said that, I still think k8s is an upgrade when you have a large team.

Re: We're Leaving Kubernetes

#27

Post by kbolino »

bhouston wrote: post #6

I also recently left Kubernetes. It was a huge waste of time and money. I've replaced it with just a series of services on Google Cloud Run and then using Google's Cloud Run Tasks services for longer running tasks. The infrastructure now incredibly understandable and simple and cost effective. Kubernetes cost us >$million in both DevOps time and actually Google Cloud costs unnecessarily, and even worse it cost us tim…

As far as I can tell, there actually is no AWS equivalent to GCP Cloud Run. The closest equivalents I know of are ECS on Fargate, which is more like managed Kubernetes except without Kubernetes compatibility or modern features, or AppRunner, which is closer in concept but also sorely lacking in comparable features.

Re: We're Leaving Kubernetes

#28

Post by nickstinemates »

loujaybee wrote: post #23
ensignavenger wrote: post #8

The article does a great job of explaining the challenges they ran into with Kubernetes, and some of the things they tried... but I feel like it drops the ball at the end by not telling us at least a little what they chose instead. The article mentions they call their new solution "Gitpod Flex" but there is nothing about what Gitpod Flex is. They said they tried microVMs and decided against them, and of course Kubern…

Yeah, that's fair. The blog was getting quite long, so we need to do some deeper dives in follow-ups. Gitpod Flex is runner-based. The runner interface is intentionally generic so that we can support different clouds, on-prem or just Linux in future. The first implemented runner is built around AWS primitives like EC2, EBS and ECS. But because of the more generic interface Gitpod now supports local / desktop environm…

Echoing the parent you're replying to. You built up all of the context and missed they payoff.

Re: We're Leaving Kubernetes

#29

Post by thenaturalist »

lmeyerov wrote: post #25

I was intrigued because the development environment problem is similar to the data scientist one - data gravity, GPU sharing, etc - but I'm confused on the solution? Oddly, I left with a funny alternate takeaway: One by one, their clever inhouse tweaks & scheduling preferences were recognized by the community and turned into standard k8s knobs So I'm back to the original question... What is fundamentally left? It sou…

> We’ll be posting a lot more about Gitpod Flex architecture in the coming weeks or months. I’d love to invite you on November the 6th to a virtual event where I’ll be giving a demo of Gitpod Flex and I’ll deep-dive into the architecture and security model at length.

Bottom of the post.

Re: We're Leaving Kubernetes

#30

Post by dikei »

xyst wrote: post #15

I do agree with the points in article that k8s is not a good fit for development environments. In my opinion, k8s is great for stable and consistent deployment/orchestration of applications. Dev environments by default are in a constant state of flux. I don’t understand the need for “cloud development environments” though. Isn’t the point of containerized apps is to avoid the need for synchronizing dev envs amongst t…

Sarcastically, CDE is one way to move cost from CAPEX (get your developer a Mac Book Pro) to OPEX (a monthly subscription that you only need to pay as long as the dev has not been lay off)

It's also much cheaper to hire contractors and give them the CDE that can be terminated on a moment notice.

Post reply on HN