Why Does Developing on Kubernetes Suck?

1. Why kubernetes for development? For most cases you don't need the whole orchestration over multiple nodes overhead, docker-compose is just fine. Incidentally, your push/pull problem is solved as you pull the dependencies and BUILD your app locally.

2. Want to have someplace to test your kube configurations? That's the ops part of devops. Set up a proper testing cluster instead of trying to cram everything in your devenv. Kube config should be in a separate project anyways.

3. No strace/bash. Look, man, this isn't even kubes' problem. That's how you build your images. Have separate dev images that use prod image as a base. Deploy dev images to testing env. Prod images to staging/prod. Build dev images locally.

4. Network debugging. That's difficult on any system. If you don't have access to the underlying host, you are screwed either way - call your devops for support. Anything else (internal ips, svc endpoints, ingresses) is trivial to debug on kube. However, we're talking about development... just don't use kube for development, period. Use docker-compose. Which, actually solves all your problems... right tools for the right job, or suffer the self-inflicted consequences.

IMHO because we are in a phase of transition.

Having worked for years in software industry , I'm convinced we are halfway to a much bigger transformation for Software Engineers / SRE , Developers etc...

I work in a Neobank ( N26 , Revolut, etc...) , we are currently in the process of re-writing our entire Core Banking System with MicroServices on top of Kubernetes with Kafka.

Not a single day pass without having engineers needing to have an exchange about defining basically all of the the terms that exist within the K8/Docker/Kafa world.

- What's a Pod ? How does a pod behave if Kafa goes down ? Do we really need ZooKeeper etc....

Their workflows is insanely complex and requires hours if not a day to deploy a single change... obviously let's not even talk about the amount of work our SRE has in the pipe to "package" the entire stack of 150+ services in K8 through a single YAML file....

I'm sure this complexity is temporary. One tech will automate all of this away and building , deploying and running will be much simpler. In it's current shape , K8 remains me of mainframe ecosystem I used to deal with at brick & mortars banks.

Powerful systems , but they requires a tremendous amount of work and experts to be properly managed and take full advantage of it. Just like mainframes , K8 leaves very little room for "mistakes" or "approximation".

One part that I agree with the author is that local kubernetes clusters or even bare metal clusters are hard. I love being able to just spin up a cluster in GKE and have it just work. It really does just work, especially for ingress and storage. Get one piece working at a time, then you can move it to its final location and do whatever else needs to happen to make that work. It's valuable to have it working in the cloud even if prod is not, since that makes for easier testing/CI.

As for your #4, it is hard, although I've seen some success with running a pod that has tcpdump in it as a sidecar. Example here: https://medium.com/@xxradar/how-to-tcpdump-effectively-in-ku... (haven't used these directions personally, but it looks very similar to what I've seen). Some networking problems will only happen in kubernetes, things like timeout configuration and nginx stuff will just be different, so it's good to be able to do this.

One other note is for logs, just use the helm charts to install kibana/fluentd/elasticsearch into your cluster. It's really easy, and then you don't have to worry about which pod is making the logs, and by default it is tagged with the pod name and other k8s metadata.

Not best-practice. Best practice is to lock down access to production to grant read-only permissions to developers and force deployments to occur through a controlled pipeline. This controlled pipeline can ensure that changes first go through a staging environment that is essentially identical to production, verify that the changes work there, and only afterwards deploy to production.

> Testing things like NetworkPolicies is also fraught.

Test, staging, pre-production environments. Particularly since the developer won't typically have the entire environment set up on their developer machine, and permissive network access is a working default, usually more restrictive network policies are setup by whoever has responsibility for the larger environment and breaks are caught in later stages of the pipeline.

> ...test ingress changes, but even then changes can take 30 minutes to take effect and can result in inscrutable error messages.

Not Kubernetes's fault. Even then it's only partially the cloud provider's fault (because of the inscrutability of said error messages) - infrastructure doesn't provision itself in a split-second and that reality stands at odds with kubectl's desire to return asynchronously, ironically enough, for a better developer experience compared to staring at a console waiting for the cloud provider to respond within a timeout.

> Maybe in the future, SSH will be as anachronistic as the floppy disk icon, but for now, I want to log in to a container, poke around, see what the state is

Oh man. Please don't. Just don't. Improve your logging and metrics first, then we'll talk.

> It’s reasonable that this image doesn’t have strace, and kind of reasonable that it doesn’t have bash, but it highlights one of the Kubernetes best practices that makes local development hard: keep your images as small as possible.

That's not a Kubernetes issue, that's a containerization issue. And a large part of the philosophy behind why it's OK in containerland is, there are only three things you need to effectively observe containerized stateless applications, and they are logs (including request tracing), exported metrics, and network traffic statistics (which really are a kind of exported metric, just not one exported by the service).

> Even if you can avoid going out to the internet when pushing an image, just building an image can take forever. Especially if you aren’t using multi stage builds, and especially if you are using special development images with extra dependencies.

So... use multi-stage image builds?

> What I want to do is just sync a file up to my pod... if your container is restarted for any reason, like if your process crashes or the pod gets evicted, you lose all of your changes.

By design. Your changes aren't going to propagate to staging, let alone production.

> In dev, I want to tail the relevant logs so I can see what I’m doing. Kubernetes doesn’t make that easy.

https://github.com/wercker/stern

> If we want to empower developers to create end-to-end full stack microservices architectures we need to provide some way to get their hands dirty with networking. Until then that last push to production will always reveal hidden networking issues.

If you need to alter multiple microservices to deliver a single feature, your architecture is probably screwed up, or your development process isn't sufficiently structured to build up individual backend services incrementally and in a serial, rollback-able fashion. You can't fix a poor architecture or SDLC by throwing tooling at the problem.