Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google admits Kubernetes container tech is too complex

theregister.com

Post reply on HN
Print view View original

271–280 of 449 posts

Re: Google admits Kubernetes container tech is too complex

#271

Post by zdw »

Ianvdl wrote: post #259
anchochilis wrote: post #221

I work on a 3-person DevOps team that just finished migrating ~20 services from GCE vms running docker-compose to GKE. It's taken us a little over a year. Partly because K8s has a steep learning curve, but also because safely transitioning services without disrupting product teams adds a lot of overhead. The investment is already yielding great returns. Developers are happy. Actual quote: "Kubernetes is the biggest q…

As someone with limited experience with containers, how does K8s allow you to move away from things like Puppet for configuration management? Does it offer some substitute that alleviates the need for something like Puppet or Ansible?

You still need something to provision the base OS and all the stuff under K8s (docker daemon, ntp, storage, networking, etc.) that it relies on, unless you go with a fully hosted solution.

Ansible or Puppet still excel at that kind of work.

Re: Google admits Kubernetes container tech is too complex

#272

Post by 0xbadcafebee »

Once I finally forced myself to learn K8s, I became depressed. I didn't think people could design something so poorly and force the whole world to adopt it. But, I guess that's another lesson: stupidity can easily hide inside complexity.

Now, can you get along wit K8s? Of course you can! Human beings have been adapting to harsh environments for hundreds of thousands of years. We can figure out some shitty software. So that means that probably, K8s will not actually improve, because people will probably adapt to it rather than change it.

So I think it's time to create a new distributed computing platform. Something with a simpler design, that has the necessary functionality built in rather than bolted on. Something that is very powerful but also terse. And of course, something that only gets complicated when you need to do something complicated, and allows you stay simple as long as possible. The ability to scale the complexity, essentially.

Re: Google admits Kubernetes container tech is too complex

#273

Post by danpalmer »

lallysingh wrote: post #249
FlyingSnake wrote: post #109

Putting on my Asbestos Longjohns: The more I look into k8s ecosystem, the more I'm convinced that it's one of those things that suits FAANG etc, but the regular Joe developer has caught on the fad and wants to add it to his repertoire, even though it's an overkill. After all no one got fired for buying IBM and recommending Kubernetes. Most teams need a simpler deployment strategies that other's have succinctly mentio…

Kubernetes solves a very real and significant problem. But before you start using it, make sure you have the problem it solves. If you're looking to have your small app eventually grow into a large one, read up on K8s and just make sure you're not blocking future-you from making your app work on it. E.g., work well in a container (which is useful for automated testing, deps management, etc), have a simple 'ping' endp…

Exactly this. Kubernetes is a service orchestrator, not a hosting platform. Those getting caught up in it just want a hosting platform, but those getting value out of it want a service orchestrator.

If you have one monolithic backend service (and most web applications really should start out this way), Kubernetes offers almost no benefits over alternatives.

Re: Google admits Kubernetes container tech is too complex

#274

Post by drewlander »

KronisLV wrote: post #155
sofixa wrote: post #79

Earlier quoted context omitted.

Docker Swarm is pretty much abandonware/on life support at best, so one should avoid using it for new stuff. Hashicorp's Nomad is the best choice on the complexity for features scale IMHO, and that's why i'm writing an article how great it is, how easier some things are and what's missing compared to Kubernetes.

And yet, Docker Compose is pretty popular for local development, so much so, that it's not uncommon to find a docker-compose.yml in the repositories for many open source projects. And Docker Swarm builds on that, by bridging the gap between Docker Compose and multi-server deployments, with tools like Swarmpit and Podman for easier management of it as well, much like Rancher does for Kubernetes. I agree that Docker Sw…

We use swarm for a small cluster in production a well. Extremely easy, zero downtime deployments are fantastic. I can explain it to someone else and quickly get them up to speed. Having said that, the fact that it seems to be on life support has made me look at other alternatives, even a simple docker-compose per node.

Re: Google admits Kubernetes container tech is too complex

#275

Post by flakiness »

je42 wrote: post #150
jillesvangurp wrote: post #97

Earlier quoted context omitted.

It's the right attitude. The management fees alone for this autopilot thing are 0.10$ per hour. Or about 70$/month. It's a bargain considering all the hidden costs kubernetes imposes in terms of requiring people that know how to tame the complexity associated with it (i.e. very expensive devops people costing magnitudes more than that). Automating those people away is worth money. I like Cloud Run for the same reason…

yep, cloud run is pretty good. unfortunately, it doesnt cover all cases. (i.e. stateful stuff like: websockets and chunking, and recurring jobs) for these case, i still have gke cluster around.

Nitpick FYI: WebSocket is coming. https://cloud.google.com/blog/products/serverless/cloud-run-...

(Still agree that Cloud Run isn't for everyone.)

Re: Google admits Kubernetes container tech is too complex

#276

Post by jeremyjh »

MaxBarraclough wrote: post #215
bob1029 wrote: post #176

I remember trying out docker sometime back in late 2013. Something about it never fully stuck with me. I always felt like the final boundary for a piece software should be the process, not the computer. Plopping an entire VM into a zipfile and saying "here's your software" felt like lazy engineering to many of us at the time (and still today). For our current stack, the answer has been to make the entire business app…

> Plopping an entire VM into a zipfile and saying "here's your software" felt like lazy engineering to many of us at the time (and still today). At the risk of nitpicking, docker images aren't the equivalent of VM images, as they don't include a kernel.

I don't know if that is the most important reason it is not equivalent. It also doesn't have any system processes; there is no systemd, sshd, no crond, etc. It doesn't need its own firewall rules configured or its own security managed. I could go on but I think you already get the point.

Re: Google admits Kubernetes container tech is too complex

#277

Post by bob1029 »

pantelisk wrote: post #184
bob1029 wrote: post #176

I remember trying out docker sometime back in late 2013. Something about it never fully stuck with me. I always felt like the final boundary for a piece software should be the process, not the computer. Plopping an entire VM into a zipfile and saying "here's your software" felt like lazy engineering to many of us at the time (and still today). For our current stack, the answer has been to make the entire business app…

I agree with you. However, after spending years and years trying to compile software that came with cryptic install instructions. Or have the author insist that since it works on their machine I 'm just doing something stupid. Docker was largely able to fix that. It's a somewhat odd solution for a too common problem, but any solution is still better than dealing with such an annoying problem. (source: made docker the…

I am totally on board with the idea of improving productivity. The issue I see is that this is avoiding a deeper problem - namely that the software stack requires a max-level wizard to set up from scratch each time.

Refactoring your application so that it can be cloned and built and ran within 2-3 keypresses is something that should be strongly considered. For us, these are the steps required to stand up an entirely new stack from source:

0. Create new Windows Server VM, and install git + .NET Core SDK.

1. Clone our repository's main branch.

2. Run dotnet build to produce a Self-Contained Deployment

3. Run the application with --console argument or install as a service.

This is literally all that is required. The application will create & migrate its internal SQLite databases automatically. There is no other software or 3rd party services which must be set up as a prerequisite. Development experience is the same, you just attach debugger via VS rather than start console or service.

We also role play putting certain types of operational intelligence into our software. We ask questions like "Can our application understand its environment regarding XYZ and respond automatically?"

Re: Google admits Kubernetes container tech is too complex

#278

Post by specialp »

Ianvdl wrote: post #259
anchochilis wrote: post #221

I work on a 3-person DevOps team that just finished migrating ~20 services from GCE vms running docker-compose to GKE. It's taken us a little over a year. Partly because K8s has a steep learning curve, but also because safely transitioning services without disrupting product teams adds a lot of overhead. The investment is already yielding great returns. Developers are happy. Actual quote: "Kubernetes is the biggest q…

As someone with limited experience with containers, how does K8s allow you to move away from things like Puppet for configuration management? Does it offer some substitute that alleviates the need for something like Puppet or Ansible?

Yes because whatever you used for app specific configuration like libraries and packages is now done in the Dockerfile and containerized. So the same thing run locally is run in the cloud. Then as far as the infrastructure for running code such as load balancers, service discovery, docker.. That is all given to you just by running K8s. So you are more concerned with shipping immutable containers to k8s than provisioning "machines".

Then you can focus on containers which can be run, tested and built wherever without the fear of broken updates or one thing stepping on another. We found back in the days of ansible and chef that we had very low confidence in upgrading hosts live. So we would then do immutable hosts and blue green deploy them to production. But why think in the scope of hosts and VMs when really you have some application that needs to run somewhere.

K8s IMO isn't the end all, I think eventually we will get to something that doesn't need containers at all and you run just processes. But it is a good step for now. Also once you have your stuff containerized it makes other non k8s stuff easy like AWS Lambda

Edit: Also yes you can use those to set up generic k8s nodes but when we ran bare metal we used kubeadm to make coreOS immutable nodes. I don't think that is used anymore haven't checked but really the best way to set up k8s is to deploy really thin hosts that have nothing but Docker and k8s. VMware and others have solutions for this too where you don't have to mess with building hosts.

Re: Google admits Kubernetes container tech is too complex

#279

Post by cameronh90 »

cyphar wrote: post #242
JeremyNT wrote: post #220

Earlier quoted context omitted.

This isn't nitpicking at all, it's an important distinction! Docker is not virtualization, it's just an abstraction that makes some Linux process isolation features easier to manage. It also allows you to bundle whatever dependencies you have in the same bundle, but that is not the same as having a VM.

Linux containers and equivalent technologies are virtualisation (specifically OS virtualisation[1]), just not a VM. Hardware virtualisation (VMs) isn't the only kind of virtualisation that exists. [1]: https://en.wikipedia.org/wiki/Operating_system-level_virtual...

By that logic, processes are arguably virtualisation too. They do after all use virtual memory.

Threads, processes and containers exist on a continuum.

Re: Google admits Kubernetes container tech is too complex

#280

Post by anchochilis »

vp8989 wrote: post #247
anchochilis wrote: post #221

I work on a 3-person DevOps team that just finished migrating ~20 services from GCE vms running docker-compose to GKE. It's taken us a little over a year. Partly because K8s has a steep learning curve, but also because safely transitioning services without disrupting product teams adds a lot of overhead. The investment is already yielding great returns. Developers are happy. Actual quote: "Kubernetes is the biggest q…

"Kubernetes is the biggest quality-of-life improvement I've experienced in my career." Can you elaborate? What exactly about Kubernetes improved the developers life?

1. Reliable rolling deployments.

We didn't have these before. Yes, you can implement them without K8s (I have, at other companies) but to get the full set of features K8s provides, such as deploying N services in parallel, taking no more than X% of your capacity offline at a time, short-circuiting in the event the app is dead on arrival, connection draining with timeouts, you end up with a VERY complicated multi-threaded codebase.

2. Seamless horizontal scale-out.

Want to scale up your app in a test environment from 3 replicas to 6 to do some performance testing? This used to be a DevOps ticket that would take a few days -- DevOps engineer tries running Terraform, but oh no, a CentOS package update seems to have broken our Puppet manifests and we have to fix that. Now the developer makes a PR to the GitOps repo where they adjust a single YAML setting.

3. GitOps/ArgoCD.

ArgoCD is possibly my favorite piece of software of all time. It provides incredible visualizations for what's happening with your Kubernetes infrastructure. It really increases your confidence and trust in the system to be able watch a deployment rollout or scaling operation happen in real time. ArgoCD makes it spectacularly obvious when something has gone wrong -- you still sometimes need to go spelunking through the GCP console or use kubectl to inspect resources, but to a much lesser degree. I cannot emphasize enough how magical it is.

These are all things that our initial implementation has delivered, we are also planning to try to leverage K8s for things like on-demand pull request preview environments, hosted developer environments, and canary deploys that are MUCH harder to implement in a world without K8s (trust me, I've done it).

Post reply on HN