Google admits Kubernetes container tech is too complex • Hacker News Board

Re: Google admits Kubernetes container tech is too complex

#181

Post by grey-area » Fri, Feb 26, 2021, 11:16 AM UTC

je42 wrote: post #164

grey-area wrote: post #116

Earlier quoted context omitted.

Thanks, that's really interesting. Everyone has different challenges and requirements, and of course different experiences. For smaller setups (say 1-10 services) I'm quite happy with cloud config and one VM per process behind one load balancer per service. It's simple to set up, scale and reproduce. This setup doesn't autoscale, but I've never really felt the need. We use Go and deploy one static binary per service…

>We use Go and deploy one static binary per service at work with minimal dependencies so docker has never been very interesting. how do you deploy your static binary to the server? (without much downtime ?)

Sorry that should have said one binary per node really, not per service (though it is one binary per service, just on a few nodes for redundancy and load).

Services behind a load balancer so one node at a time replaced then restarted behind that, and/or you can do graceful restarts. There are a few ways.

They're run as systemd units and of course could restart for other reasons (OS Update, crash, OOM, hardware swapped out by host) - haven't noticed any problems related to that or deploys and I imagine the story is the same for other methods of running services (e.g. docker). As there is a load balancer individual nodes going down for a short time doesn't matter much.

Re: Google admits Kubernetes container tech is too complex

#182

Post by kleiba » Fri, Feb 26, 2021, 11:17 AM UTC

kleiba wrote: post #107

As someone who has never worked with containers, I find this rather amusing: Kubernetes is so complex that Google needs to roll out an "autopilot" feature, yet "it has won in the critically important container orchestration space". Makes you wonder what the alternatives must have looked like?!

Could the downvoter please explain?

Re: Google admits Kubernetes container tech is too complex

#183

Post by grey-area » Fri, Feb 26, 2021, 11:18 AM UTC

te_chris wrote: post #141

grey-area wrote: post #123

Earlier quoted context omitted.

Standardised declarative config/setup I've found really useful, declarative resource specification (as in n servers for service y) I don't really have a use for though I could see how in larger companies with hundreds of servers and lots of employees it would be attractive. If it is rock solid and reliable and impossible to mess up, I could see it being really nice (as your experiences sounds), though it feels to me…

I mean yeah, I’m not arguing against autopilot and would probably have used it if it existed when we kicked off. Cloud Run had just come out and I really wanted to use that but the networking wouldn’t do what we needed, so had to go all in. But in the end, we learned enough Kube, we specified it all with Terraform, we haven’t tried to do too many fancy or complicated things, and we have reliable infrastructure fed wi…

Sounds like a nice reliable experience to me, thanks for sharing your experience on it.

If I hear more reports like this I might just have to try out GKE :)

Re: Google admits Kubernetes container tech is too complex

#184

Post by pantelisk » Fri, Feb 26, 2021, 11:20 AM UTC

bob1029 wrote: post #176

I remember trying out docker sometime back in late 2013. Something about it never fully stuck with me. I always felt like the final boundary for a piece software should be the process, not the computer. Plopping an entire VM into a zipfile and saying "here's your software" felt like lazy engineering to many of us at the time (and still today). For our current stack, the answer has been to make the entire business app…

I agree with you. However, after spending years and years trying to compile software that came with cryptic install instructions. Or have the author insist that since it works on their machine I 'm just doing something stupid. Docker was largely able to fix that.

It's a somewhat odd solution for a too common problem, but any solution is still better than dealing with such an annoying problem. (source: made docker the de facto cross-teams communication standard in my company. "I 'll just give you a docker container, no need to fight trying to get the correct version of nvidia-smi to work on your machine" type of thing)

It probably depends on the space and types of software you 're working on. If it's frontend applications for example then its overkill. But if somebody wants you to let's say install multiple elasticsearch versions + some global binaries for some reason + a bunch of different gpu drivers on your machine (you get the idea), then docker is a big net positive. Both for getting something to compile without drama and for not polluting your host OS (or VM) with conflicting software packages.

Re: Google admits Kubernetes container tech is too complex

#185

Post by secondcoming » Fri, Feb 26, 2021, 11:21 AM UTC

endymi0n wrote: post #71

fen4o wrote: post #13

I understand their rationale. We manage thousand Kubernetes clusters and end-users can find lots and lots of creative way to shoot themselves in the foot: - I can store anything in a secret? Let's have thousands of cat images. Etcd then stops working because we have over 2GB of funny cats in the key store. - I can run a root Pod? Lets mount the docker socket and start building images with it. Oh and by the way, I nev…

I fully agree with your points and would sum them up as "Kubernetes has a steep learning curve, a (quite) large interface and ample opportunities to shoot yourself into the foot with it" (plus, they're very funny). However playing the devil's advocate here: If you actually took the steps of learning the basic abstractions, then for me it's really hard to see what you could still get rid of. If you actually go all-in…

Ok, but we have a similar setup that runs on GCE instances. Deploying involves building an image and pushing a button. We don't really have the need for an Infrastructure team.

Re: Google admits Kubernetes container tech is too complex

#186

Post by fragile_frogs » Fri, Feb 26, 2021, 11:22 AM UTC

FlyingSnake wrote: post #109

Putting on my Asbestos Longjohns: The more I look into k8s ecosystem, the more I'm convinced that it's one of those things that suits FAANG etc, but the regular Joe developer has caught on the fad and wants to add it to his repertoire, even though it's an overkill. After all no one got fired for buying IBM and recommending Kubernetes. Most teams need a simpler deployment strategies that other's have succinctly mentio…

That reminds me of Basecamps article about "The majestic monolith" https://m.signalvnoise.com/the-majestic-monolith/

Re: Google admits Kubernetes container tech is too complex

#187

Post by qeternity » Fri, Feb 26, 2021, 11:28 AM UTC

I'm surprised to see this k8s consensus on HN. Kuberentes has a very steep learning curve. I spent a week over Xmas 2019 just bringing up and tearing down clusters until I got something that was functional. The documentation is not particularly friendly so there was a lot of trial and error. But eventually it clicked. Then a couple months later we decided to invest by slowly moving non production services over, and then ultimately moving everything over.

We have not had any major k8s incidents in the past 12+ months of running our own HA cluster. We even run HA Postgres using local volumes. We've pretty much moved everything over and couldn't be happier.

We have red teamed disaster scenarios, we can bring the full cluster up on another cloud provider using vanilla k8s in about an hour (including a restore from Postgres S3 backups + wals). And this is all with a very small team.

Re: Google admits Kubernetes container tech is too complex

#188

Post by williesleg » Fri, Feb 26, 2021, 11:41 AM UTC

No shit sherlock.

Re: Google admits Kubernetes container tech is too complex

#189

Post by jononomo » Fri, Feb 26, 2021, 11:42 AM UTC

YawningAngel wrote: post #173

machiaweliczny wrote: post #158

Earlier quoted context omitted.

IMO fixing BEAM VM to easily work on cluster could be better for distributed systems than k8s.

That would do nothing to help the 99% of users who don't use BEAM VM languages

They'd have a reason to start using BEAM VM languages, though.

Re: Google admits Kubernetes container tech is too complex

#190

Post by notyourday » Fri, Feb 26, 2021, 11:45 AM UTC

justsomeuser wrote: post #138

trinovantes wrote: post #36

Kubernetes has to be most complex software I've ever tried to learn. I eventually gave up and decided to stick with simple single machine docker-compose deployments. I figure by the time any of my personal projects actually need to scale beyond 1 machine, I'd probably have enough revenue that I can afford to hire someone else to worry about it.

I’m leaning towards using Ubuntu/systemd to start all my services, and using a single container per project with SQLite. This way: - I can easily move from dev to prod by using the private container registry - I have apt-get on the server and just use the default Ubuntu - No distributed/network state I used to use Core OS, but all these container OSes are here today and gone tomorrow, and normally have their own conf…

systemd-nspawn is the most underrated piece of software on any modern link system. It just works and does exactly what it needs to do. Not to mention using a directory structure as a container image?! What is this sorcery?!