Google admits Kubernetes container tech is too complex • Hacker News Board

Re: Google admits Kubernetes container tech is too complex

#141

Post by te_chris » Fri, Feb 26, 2021, 9:59 AM UTC

grey-area wrote: post #123

te_chris wrote: post #80

Earlier quoted context omitted.

No, but it’s not just complexity. We’re a small company, but after spending about a month to setup our GKE a couple of years ago (incl learning terraform in the process), it’s been rock solid, low involvement and reliable. Declarative resource specification is a legitimate game changer and you couldn’t pay me to go back.

Standardised declarative config/setup I've found really useful, declarative resource specification (as in n servers for service y) I don't really have a use for though I could see how in larger companies with hundreds of servers and lots of employees it would be attractive. If it is rock solid and reliable and impossible to mess up, I could see it being really nice (as your experiences sounds), though it feels to me…

I mean yeah, I’m not arguing against autopilot and would probably have used it if it existed when we kicked off. Cloud Run had just come out and I really wanted to use that but the networking wouldn’t do what we needed, so had to go all in. But in the end, we learned enough Kube, we specified it all with Terraform, we haven’t tried to do too many fancy or complicated things, and we have reliable infrastructure fed with a simple deployment/build pipeline (just using cloud build, it’s clunky but it works). I mean maybe I’m conflicted as we’re part of GCS, but the system works well for us. If it didn’t we could’ve jumped to Azure or AWS and got credits there.

Re: Google admits Kubernetes container tech is too complex

#142

Post by Seb-C » Fri, Feb 26, 2021, 10:03 AM UTC

trinovantes wrote: post #36

Kubernetes has to be most complex software I've ever tried to learn. I eventually gave up and decided to stick with simple single machine docker-compose deployments. I figure by the time any of my personal projects actually need to scale beyond 1 machine, I'd probably have enough revenue that I can afford to hire someone else to worry about it.

This is a honest question, but may I ask what you found to be hard about it?

For long I was scared of it because so many people say it's crazy complex. But actually it took no more than a dozen of hours for me to learn it and get a working setup on aws.

Maybe being full stack and having a strong knowledge of Linux and Docker helped.

Now I'm not pretending to be an expert with it, and there are certainly traps and mistakes that I didn't experience yet. But I don't understand what people find to be hard about it.

Re: Google admits Kubernetes container tech is too complex

#143

Post by nabki » Fri, Feb 26, 2021, 10:03 AM UTC

Fundementally, the problems that lead to all these solutions is that there isn't any way to abstract (at a low level) the differences between horizontal and vertical scaling of computing power. You will always have to care about the difference between scaling a node and scaling nodes when developing any sort of web based software stack. It doesn't matter if you're running VMs or containers, or are on the cloud or serverless. The fact that you can't abstract away the differences of horizontal vs vertical scaling will always dictate how you design the architecture of your service, all the way down to the coding patterns you'll follow. Thats really the problem. K8 and containers are solving a symptom of this problem, and not the problem itself.

Re: Google admits Kubernetes container tech is too complex

#144

Post by KronisLV » Fri, Feb 26, 2021, 10:04 AM UTC

dilyevsky wrote: post #88

fen4o wrote: post #13

I understand their rationale. We manage thousand Kubernetes clusters and end-users can find lots and lots of creative way to shoot themselves in the foot: - I can store anything in a secret? Let's have thousands of cat images. Etcd then stops working because we have over 2GB of funny cats in the key store. - I can run a root Pod? Lets mount the docker socket and start building images with it. Oh and by the way, I nev…

I once misconfigured iptables and locked myself out of our buildserver. Had to call lab support in a different country. Is Linux too complicated? Joyent famously took down their whole region by rebooting wrong nodes. It’s almost like running distributed networks of supercomputers at scale is hard or something...

> Joyent famously took down their whole region by rebooting wrong nodes.

In case anyone's interested, here's a pretty funny and educational talk by Bryan Cantrill about that particular incident:

GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind

https://www.youtube.com/watch?v=30jNsCVLpAE

Re: Google admits Kubernetes container tech is too complex

#145

Post by konjin » Fri, Feb 26, 2021, 10:04 AM UTC

collyw wrote: post #127

Clewza313 wrote: post #32

Earlier quoted context omitted.

> But it is a surprisingly negative headline. That's the Register's schtick, they're snarky about everything.

They are usually closer to the truth than evangelists telling us how this new tool is going to solve all our problems.

Reality is too negative for social media.

Re: Google admits Kubernetes container tech is too complex

#146

Post by orthoxerox » Fri, Feb 26, 2021, 10:15 AM UTC

HenryKissinger wrote: post #15

trestenhortz wrote: post #12

This was glaringly obvious from day one. Containers add vast complexity, add another layer of complexity on top.

Can you explain the whole concept of Kubernetes to someone whose knowledge of computers is limited to making simple webpages with HTML and using Excel/VBA?

You know how there are linked references in Excel files that refer to other Excel files? What if you want to keep multiple copies of your Excel file with macros that refer to different files with data? And so you can run them on Windows, Mac or in your browser?

Kubernetes basically lets you define your references not as "c:\jon\reports\fy2020_final_final_2_comments_review_Bob_final.xlsx", but as "fy-report", with "fy-report" being defined elsewhere.

This is required to help you run the same program in different circumstances without breaking everything. You can say "run it on this slow computer with this test data", or you can "run it on many big computers with real data", but the program is exactly the same.

What makes it so complicated is that Kubernetes tries to abstract everything a given program would need, so it's not just references to external files, but practically the whole computer with all its network connections that must be defined elsewhere using Kubernetes' special language.

A lot of this special language is the same for 99% of programs, like in Excel you want your VLOOKUP to work on a $-pinned range with the last parameter set to FALSE 99% of the time. This makes people make the same stupid mistakes and finding them is hard.

And of course, this special language means you have to relearn a lot you know about running programs on computers, like when you move from Excel formulas to VBA.

Re: Google admits Kubernetes container tech is too complex

#147

Post by einpoklum » Fri, Feb 26, 2021, 10:15 AM UTC

trinovantes wrote: post #36

Kubernetes has to be most complex software I've ever tried to learn. I eventually gave up and decided to stick with simple single machine docker-compose deployments. I figure by the time any of my personal projects actually need to scale beyond 1 machine, I'd probably have enough revenue that I can afford to hire someone else to worry about it.

> simple ... docker-compose deployments

So, not simple.

Re: Google admits Kubernetes container tech is too complex

#148

Post by diarrhea » Fri, Feb 26, 2021, 10:22 AM UTC

FlyingSnake wrote: post #109

Putting on my Asbestos Longjohns: The more I look into k8s ecosystem, the more I'm convinced that it's one of those things that suits FAANG etc, but the regular Joe developer has caught on the fad and wants to add it to his repertoire, even though it's an overkill. After all no one got fired for buying IBM and recommending Kubernetes. Most teams need a simpler deployment strategies that other's have succinctly mentio…

A friend of mine is a contributor to k8s itself, and of course, this all comes incredibly easy to them. Following their recommendation, I gave it a shot for my single-person, single-node (!) homelab, all without using MicroK8s, k3s or similar.

After a week of almost full-time work, I threw in the towel. Admittedly, I also had to learn concepts like reverse proxies alongside, too, so I was by no means well-equipped to begin with.

Yet, tossing together some docker-compose.yml files and "managing" them with a Python script has worked very well. Kubernetes really scarred me in that sense, and I am healed! Also, Caddy has helped me in actually enjoying configuring the webserver.

Re: Google admits Kubernetes container tech is too complex

#149

Post by LeonM » Fri, Feb 26, 2021, 10:24 AM UTC

I have a love/hate relation with K8S, I have several customer project that I work on deployed on GKE. For the most part it has been a rock solid set-and-forget experience, but...

The problem for me is that these project are not under full time development. Most of them see maybe one or two deployments per year. It seems like every time I do a deployment, some part of the YAML config structure has been deprecated, with no clear path on how to migrate. In the end I have to spend a day figuring out how to rewrite the YAML configs to do _exactly the same_ as before. This is really hard to explain to a customer.

But it's not just K8S itself that suffers from this, third party components do this too. Stuff like ingress control is a nightmare, I've seen the nginx ingress controller (not to be confused with the 'other' nginx ingress controller) grow from a 50 LOC base config to multiple KLOC of YAML config you are supposed to pipe to your production environment. The ACME/LE extension (cert-manager) is even worse, the default config is 26KLOC long! [0]. And you are supposed to pipe this config straight into your environment. The amount of new and amazingly complex components that are added with each new release is staggering.

If you are an average Joe developer and you want to use containers in production, just stick with a machine with docker-compose on it. It's a lot easier to maintain and has far fewer surprises down the road. It is also much easier to make a quote on maintaining a VPS box that having to get a crystal ball to predict when Google will deprecate parts of your setup.

[0] https://github.com/jetstack/cert-manager/releases/download/v...

Re: Google admits Kubernetes container tech is too complex

#150

Post by je42 » Fri, Feb 26, 2021, 10:27 AM UTC

jillesvangurp wrote: post #97

trinovantes wrote: post #36

Kubernetes has to be most complex software I've ever tried to learn. I eventually gave up and decided to stick with simple single machine docker-compose deployments. I figure by the time any of my personal projects actually need to scale beyond 1 machine, I'd probably have enough revenue that I can afford to hire someone else to worry about it.

It's the right attitude. The management fees alone for this autopilot thing are 0.10$ per hour. Or about 70$/month. It's a bargain considering all the hidden costs kubernetes imposes in terms of requiring people that know how to tame the complexity associated with it (i.e. very expensive devops people costing magnitudes more than that). Automating those people away is worth money. I like Cloud Run for the same reason…

yep, cloud run is pretty good. unfortunately, it doesnt cover all cases. (i.e. stateful stuff like: websockets and chunking, and recurring jobs)

for these case, i still have gke cluster around.