I've found https://www.telepresence.io/ to be helpful. It allows to integrate a local running process into a K8s cluster seamlessly allowing for fast iteration and easy debugging.
Why Does Developing on Kubernetes Suck?
21–30 of 87 posts
Re: Why Does Developing on Kubernetes Suck?
#22I think the author is doing it wrong personally. Just spin up a real dev cluster and use that? Why putz around with setups on your laptop? That’s never going to scale team-wise? And if you damn well insist on running your own local k8s cluster for development then the whole thing should be automated infrastructure as code anyways that you start with a command so that other developers will get the same behavior. Even…
Ability to run locally a production like setup (at smaller scale of course) is a big plus imho. 1st, you know how it works, 2nd smaller iterations/feedaback cycles (restart locally vs. push and wait for tests to run).
I agree with this. But usually you have both. A staging/test environment is nice, but if you have 5 developers and one test env then devs will quickly end up in line waiting for the test env to free up.
Remote debugging is also hard. Much easier when running locally.
Re: Why Does Developing on Kubernetes Suck?
#23If iterations take longer than 10s, then I'm really not interested. Typical Docker based CIs are ten minutes to build. Nevermind the deployment. A ridiculous waste of time and energy.
If you have a fleet of services and currently work on services A and B you can just start up A - G in docker-compose based on CI-built images but mount your local build output into the A and B containers. Override their commands to watch filesystem changes so they reload themselves when you build.
This can be easily managed with an personal override file on top of a committed compose file used by your team.
Re: Why Does Developing on Kubernetes Suck?
#24If iterations take longer than 10s, then I'm really not interested. Typical Docker based CIs are ten minutes to build. Nevermind the deployment. A ridiculous waste of time and energy.
I refuse to do work on docker because of this.
If you’re working with python or javascript or something similar, the usual solution is to mount a directory from your host OS as a docker volume, so that you don’t have to rebuild image layers on every change. Isn’t it?
Re: Why Does Developing on Kubernetes Suck?
#25If iterations take longer than 10s, then I'm really not interested. Typical Docker based CIs are ten minutes to build. Nevermind the deployment. A ridiculous waste of time and energy.
If you have a fleet of services and currently work on services A and B you can just start up A - G in docker-compose based on CI-built images but mount your local build output into the A and B containers. Override their commands to watch filesystem changes so they reload themselves when you build. This can be easily managed with an personal override file on top of a committed compose file used by your team.
I agree the local dev environment UX is pretty much there. I use volume mounts myself.
But when you're deploying on a remote host. OMG.
Just sharing a Docker image internally is anguish.
Developing as a remote team using Docker / Kubernetes workflow is just crazy. You're way WAY better off using serverless, where some implementations take ~2s to deploy my Go binary.
Re: Why Does Developing on Kubernetes Suck?
#26> Why Does Developing on Kubernetes Suck ? IMHO because we are in a phase of transition. Having worked for years in software industry , I'm convinced we are halfway to a much bigger transformation for Software Engineers / SRE , Developers etc... I work in a Neobank ( N26 , Revolut, etc...) , we are currently in the process of re-writing our entire Core Banking System with MicroServices on top of Kubernetes with Kafka…
There is a lot of terminology for sure. We've gone from "is docker ready for production usage?" to "Hey Bob, look at this, a service which isn't running in a container yet" in about 5 years.
In order to work effectively within kubernetes you've got to understand docker, with all of its nomenclature (images, containers, volumes, layers etc). Then you've got the things kubernetes layers on top (pods, ingress, stateful sets, persistent volumes etc).
You then probably layer Helm (adding tiller, deployments, charts, templates) or something else on top of that.
We've still got the complexity from before that, remember when ansible, puppet and chef were all the rage? You're probably still using that to initially configure your bare metal.
You still have to understand all of the quirks of your data store, except now instead of throwing everything in a huge SQL DB you have some things in Redis, MongoDB, Kafka and maybe some legacy stuff in that big DB in the corner.
Let's not forget your CI stack in Jenkins, Travis or maybe codefresh which builds and runs your tests. That probably does some magic with docker too.
Oh, and if anything goes wrong in any of that you'd better be well grounded in general unixy debugging (permissions, networking etc).
It's certainly getting to the point that I feel no one person can be an expert in all of it, especially because when it all works you can more or less let it do its thing. When it breaks though it feels like playing Jenga.
Re: Why Does Developing on Kubernetes Suck?
#27What would you recommend for development of some private project on a local server? No cloud cluster, bare metal and great opportunity to learn. Something on the order of 20-40 services for even the most trivial things. Development speed above all else
Tilt (the product created by the company hosted by the blog) is an implied solution.
There is also Draft (Microsoft), Skaffold (Google) and Garden.io
Here is comparison of the last 3 https://codefresh.io/howtos/local-k8s-draft-skaffold-garden/
(I have no affiliation with any of those solutions, I just co-authored the comparison article)
Re: Why Does Developing on Kubernetes Suck?
#28Ok, wow. Talk about mixed feelings towards the article... the author is clearly knowledgeable, but it is so unclear why in the world does he opt for this kind of torture. 1. Why kubernetes for development? For most cases you don't need the whole orchestration over multiple nodes overhead, docker-compose is just fine. Incidentally, your push/pull problem is solved as you pull the dependencies and BUILD your app locall…
> 1. Why kubernetes for development? For most cases you don't need the whole orchestration over multiple nodes overhead, docker-compose is just fine. Incidentally, your push/pull problem is solved as you pull the dependencies and BUILD your app locally. Then it's not hermetic and repeatable. We strive for reproducibility between envs, because it reduces the chance that we're relying on incorrect assumptions about env…
> Then it's not hermetic and repeatable.
in my experience, the core difference between local env and production in kubernetes is configuration of baremetal nodes and the firewall between them. In essence, if your docker-compose does not cover the logical equivalent of your cluster... well, you have architecture problems, and you should probably not solve them just by throwing kubernetes at it. Either that, or, covering your next point
> Is your test env just for you, or is there a chance you could break other engineers?
if your system requires significantly more resources or for some other reason you just can not bring the whole system up via docker-compose and you just really need to have a staging-level-thing-in-development, build a separate cluster for development. Leverage kubernetes namespaces so that each developer could have his/her own little namespace-sandbox-kube. I've seen that working pretty well in practice.
> There should be a means of injecting tooling
i agree that https://github.com/kubernetes/kubernetes/issues/30656 should have been closed long ago, but, again, this is not something kube should (or could) solve. Consider the simple example of your base images - one is using alpine, the other - debian. Injecting tooling is very different on these two. This could be in theory solved by injecting statically-linked executables, but, boy, this is another can of worms i'd rather leave in the hands of end-users...
> It's dangerous to get away from a real production config.
you will get away from a real production config. Your actual cluster nodes have a different configuration already. And, in my experience, that is more of an issue regarding network than additional ingress smarts. And, again, you can run a local version of the ingress as well if you need to replicate its' behavior (ingress is usually just managed nginx anyway, if you're not into exotics like Kong)... And if your control planes' smarts are undocumented and really require a full blown grpc-ingress with all the bells and whistles that kube brings, well... something's rotten in Denmark then...
Re: Why Does Developing on Kubernetes Suck?
#29Ok, wow. Talk about mixed feelings towards the article... the author is clearly knowledgeable, but it is so unclear why in the world does he opt for this kind of torture. 1. Why kubernetes for development? For most cases you don't need the whole orchestration over multiple nodes overhead, docker-compose is just fine. Incidentally, your push/pull problem is solved as you pull the dependencies and BUILD your app locall…
I agree with you that many of these are self inflicted problems. I particularly like your answer to #3, which I think I'm going to have to start adopting! One part that I agree with the author is that local kubernetes clusters or even bare metal clusters are hard. I love being able to just spin up a cluster in GKE and have it just work. It really does just work, especially for ingress and storage. Get one piece worki…
thank you. As for #4, i've heard good things about https://istio.io - supposedly it does provide nice means to gain introspection to the traffic between services.
Re: Why Does Developing on Kubernetes Suck?
#30Ok, wow. Talk about mixed feelings towards the article... the author is clearly knowledgeable, but it is so unclear why in the world does he opt for this kind of torture. 1. Why kubernetes for development? For most cases you don't need the whole orchestration over multiple nodes overhead, docker-compose is just fine. Incidentally, your push/pull problem is solved as you pull the dependencies and BUILD your app locall…
1. Kubernetes is when your service needs google kind of load which 90% of systems don't. Just trade it for simple VM or use LXD containers. Don't jump on next hype cycle. If something works for Google does not mean will work for you.
2. Don't spend too much on it just use traditional knowledge and, you can still use bare-metal, VM or LXD container. Focus on application not on programming a tool designed to solve problems for Google size deployment.
3. Kubernetes is a completely different way of doing things. So if you want to leverage old cluster and distributed system knowledge you can use VM or LXD containers. Kubernetes is an overkill.
4. Abandon kubernetes instead of complaining, just use simple systems.