Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Does Developing on Kubernetes Suck?

blog.tilt.dev

Post reply on HN
Print view View original

61–70 of 87 posts

Re: Why Does Developing on Kubernetes Suck?

#61

Post by merb »

cbanek wrote: post #6
arpa wrote: post #2

Ok, wow. Talk about mixed feelings towards the article... the author is clearly knowledgeable, but it is so unclear why in the world does he opt for this kind of torture. 1. Why kubernetes for development? For most cases you don't need the whole orchestration over multiple nodes overhead, docker-compose is just fine. Incidentally, your push/pull problem is solved as you pull the dependencies and BUILD your app locall…

I agree with you that many of these are self inflicted problems. I particularly like your answer to #3, which I think I'm going to have to start adopting! One part that I agree with the author is that local kubernetes clusters or even bare metal clusters are hard. I love being able to just spin up a cluster in GKE and have it just work. It really does just work, especially for ingress and storage. Get one piece worki…

> that local kubernetes clusters or even bare metal clusters are hard

what's so hard about k3s, kubespray or minikube? the only problem k8s has, that without a linux dev machine you are going through the vm pain. but anything else is just simple.

btw. using mac sucks when trying to develop with local k8s. windows is better/okish, but it's best to run directly on linux.

Re: Why Does Developing on Kubernetes Suck?

#62

Post by zsoltk »

tutfbhuf wrote: post #53
dragonsh wrote: post #52

Earlier quoted context omitted.

> Machine count corresponds to load; abstraction doesn't. K8s is developed to manage large number of service nodes running docker or oci containers coming from Google borg project. So it was designed for managing google level of service loads. It doesn't help when a service is only for few million users with few hundred or thousand containers. LXD and traditional tools can already do it without learning new way with…

> It doesn't help when a service is only for few million users with few hundred or thousand containers. Well my company switched from VMs to k8s (few hundred containers) and it helped. Proof me wrong.

My company switched from VMs to a managed k8s cluster (gcloud). It made everything harder, the application became about 5x slower and instead of a $300 monthly bill we paid for the VMs now we pay $1300.

So in summary:

* we still have same amount of developers

* we still need same amount of sysadmins

* we introduced more complexity

* the application became slower

* the whole thing became much more expensive (managed k8s price, education, consulting, etc.)

* but! we have the same amount of income every month. Surprisingly our users don't pay more for the same service because we run our stuff on k8s

Re: Why Does Developing on Kubernetes Suck?

#63

Post by oblio »

tutfbhuf wrote: post #60
oblio wrote: post #55

Earlier quoted context omitted.

Do you manage Kubernetes, personally? Did you manage the VMs?

Yes, i'm a DevOps engineer. I have to say, we use the Google Cloud and many Google Services, this makes live so much easier. Before that we managed everything ourselves on bare metal with debian VMs.

Not exactly my point. Are you comparing managing bare metal VMs with using a managed Kubernetes cluster? That's hardly a fair comparison.

Re: Why Does Developing on Kubernetes Suck?

#64

Post by orweis »

With all the love for K8s - this post is spot on in many points.

Especially around accessing containers and observability. But ... That's where modern solutions of virtual-logging come in: e.g https://Rookout.com

When your software connects back to you and you can instrument it on the fly (Add logs line, non-breaking-breakpoints, etc.) most of these pain points are resolved

Re: Why Does Developing on Kubernetes Suck?

#65

Post by cassianoleal »

merb wrote: post #61
cbanek wrote: post #6

Earlier quoted context omitted.

I agree with you that many of these are self inflicted problems. I particularly like your answer to #3, which I think I'm going to have to start adopting! One part that I agree with the author is that local kubernetes clusters or even bare metal clusters are hard. I love being able to just spin up a cluster in GKE and have it just work. It really does just work, especially for ingress and storage. Get one piece worki…

> that local kubernetes clusters or even bare metal clusters are hard what's so hard about k3s, kubespray or minikube? the only problem k8s has, that without a linux dev machine you are going through the vm pain. but anything else is just simple. btw. using mac sucks when trying to develop with local k8s. windows is better/okish, but it's best to run directly on linux.

Why is it worse on the mac than on Windows? Legit question here, I haven't run Windows in many years so wouldn't know.

I find developing on k8s on the mac just fine. Most of the time I either just run the Docker for Desktop k8s feature, or fire up a kind cluster.

kind fits really well with VS Code's Remote - Containers extension for a completely isolated, all batteries included dev experience.

Re: Why Does Developing on Kubernetes Suck?

#66

Post by majewsky »

the8472 wrote: post #45

The one thing I have found tedious with k8s clusters is when you need to work on the host system in relation to a specific pod. E.g. to diagnose kernel issues or for profiling/debugging tools that need unrestricted root. This can require jumping through several hoops. Find the node, get through some bastion host, inject SSH keys into target node, build an SSH chain to get into the host, then escalate from admin to ro…

Run a privileged container with the host's procfs mounted into it via hostPath volume. Then you can use nsenter to break out of the container into the host system.

Re: Why Does Developing on Kubernetes Suck?

#67

Post by tinco »

sametmax wrote: post #58
tinco wrote: post #49

Earlier quoted context omitted.

There's a difference between going offline at a set time for a set amount of time, and going offline randomly for unpredictable amounts of time. HA is not just about getting that sixth 9, it's also about the peace of mind that you are insulated from a range of problems. Not only would I not worry about our SQL service going down, I also don't have to worry about what I have to do when a node goes down, because that's…

Even an occasional site crash is ok. I'm not saying it's a good user xp, and avoiding it is both better for the tech team and the marketting one. But the cost of a 500 is always to be balanced with the cost of going full throttle with micro services. As perfectionists, it's easy to fall into the trap of choosing the later by default.

What do micro services have to do with it? If anything, HA is harder to achieve with microservices. There's all these weird false dichotomies going around in this thread.

Re: Why Does Developing on Kubernetes Suck?

#68

Post by dragonsh »

tutfbhuf wrote: post #53
dragonsh wrote: post #52

Earlier quoted context omitted.

> Machine count corresponds to load; abstraction doesn't. K8s is developed to manage large number of service nodes running docker or oci containers coming from Google borg project. So it was designed for managing google level of service loads. It doesn't help when a service is only for few million users with few hundred or thousand containers. LXD and traditional tools can already do it without learning new way with…

> It doesn't help when a service is only for few million users with few hundred or thousand containers. Well my company switched from VMs to k8s (few hundred containers) and it helped. Proof me wrong.

Cannot comment on it unless see it in details that adding additional moving components of k8s helped.

Re: Why Does Developing on Kubernetes Suck?

#69

Post by the8472 »

majewsky wrote: post #66
the8472 wrote: post #45

The one thing I have found tedious with k8s clusters is when you need to work on the host system in relation to a specific pod. E.g. to diagnose kernel issues or for profiling/debugging tools that need unrestricted root. This can require jumping through several hoops. Find the node, get through some bastion host, inject SSH keys into target node, build an SSH chain to get into the host, then escalate from admin to ro…

Run a privileged container with the host's procfs mounted into it via hostPath volume. Then you can use nsenter to break out of the container into the host system.

I already used a host mount to inject SSH keys. I guess I can just skip the rest and automate the steps to pivot fully into the host.

Re: Why Does Developing on Kubernetes Suck?

#70

Post by metzby »

aeyes wrote: post #17

The only time I develop on Kubernetes is when I develop something which builds on top of the Kubernetes API. In that case I use skaffold. For all the rest Docker Compose is a better fit. You don't develop your full CI pipeline on your machine either. So why care about the orchestration?

I'd love to talk if you use skaffold; I (disclaimer: building it) work on Tilt, and suspect it's a better fit. Especially because it can work on either k8s, like skaffold, or docker-compose, with a useful UI on top.
Post reply on HN