Will Kubernetes Collapse Under the Weight of Its Complexity?

Yeah, thanks for clarification. As a technical person I find myself a bit confused as I don't know if I should follow some buzz and try to make money in the short term of go deep and go for the long term. Will it pay off ?

Of course I get more satisfaction in "feeling" a "deep" and "true" technical folk.

This is my personal experience on the matter as a DevOps consultant who periodically interviews in the traditional manner (as opposed to getting gigs from people that already worked with me).

Despite having 15+ years of *nix experience, including internals, having a track record of building large scalable infra and knowing a few different programming languages, what happened to me was this: I was getting filtered out because I didn't have Docker and Kubernetes and even (at one point) Cloudformation and/or Terraform. No problem - I learned those things (minus Kubernetes, so far) quite quickly. Much more quickly than the grueling trial-by-fire years of Unix administration. I like to know how things work, not just how to use them.

So if you wonder and worry about the state of enterprise IT some days, look no further than hiring managers themselves, who will pick a 25 year-old who writes YAML for some abstraction-of-an-abstraction system that does infrastructure under the hood, infrastructure that people kind of don't really try very hard to understand. After all, it's disposable thanks to infrastructure-as-code, right?

How do I know this? Well, I've seen shop after shop that's suffered a spaghetti infrastructure, using all the latest and greatest, from AWS and Kubernetes and Docker and other abstraction layers above AWS. And what happens is that it gets so complex that no one knows what's really going on, and at the very least two common symptoms arise: people are terrified during releases and they take hours, with many people on a call together very late at night; they spend a fortune on extra instances (in the case of AWS) because they haven't properly worked out environment separations (they had trouble keeping them the same, or one of many other problems).

A talented dev manager I used to work with used to complain that they had trouble hiring people who knew Javascript well, but they had expert after expert of some fancy JS framework try to interview, unable to answer the fundamentals-types of questions. I think it's similar with enterprise infrastructure.

I don't know what to say. I hope things go full swing and people who know how things work under the hood can charge consulting dollars for fixing the fuckups. It's not enough to know YAML, you also need to have wisdom in maintaining complex infrastructure, understand the delicate balance between change and stability, and be able to troubleshoot when it goes wrong WITHOUT just 'rinsing and repeating' where you learn absolutely no lessons at all.

[edit:] One theory for all of this. Some of the big shops (Google, FB, Netflix, etc.) did it right, and now everyone is trying to copy the style of infrastructure management, except doesn't have the talent or wisdom to do it well.

Yeah, it is. But that wasn't the question. MS Office SaaS or Adobe Creative Cloud is expensive too. AWS/GCloud is already quite pricey as it is, and every cloudified thing they roll out is just a money pump for people who don't know better.

But if you want to roll your own, then you can, it's FOSS after all. kubeadm works very well.

Comparing Synology with anything Google is silly, but useful. You can rent dedicated machines easily. Leaseweb is nice. And then you can build your infrastructure for cheaper.

> money in the short term [or] go deep and go for the long term

I think these are never contradictory, at least I think software jobs allow for the flexibility to keep working on high-yield things while learning something for depth on the side. (Or vice-versa, if you live off your side job, and do/learn something deeper/longterm.)

Especially with network, virtualization, security, distributed systems (DB, storage), etc. the ideas that are trendy now are spin-offs of older things. (Docker is basically LXC with a sane UI/UX, which is just chroot done better, a'la BSD jails, which is just Solaris zones, which is just IBM mainframe partition shit. Similarly everything on the network, like overlay networking, VXLAN, Geneve, OpenFlow, etc. are just natural layering abstractions and applied trade offs that were considered silly a decade ago, or were simply not needed a decade ago. Now that north-south traffic is dwarfed by east-west traffic in every system [this means that north is the client, south is your "DB", and east-west just means services in your system, and since people nowadays use a lot more complicated systems, the inter-component = intra-system communication is huge, compared to the few MBs sent to the client].)

As someone that has stared at a number of those projects, at this point I'm pretty excited that there are a number of different alternatives which tweak different knobs in the solution space of the problem. I think in the next few years you'll see a winnowing of choices in each category, and problem bundling of the top tier choices into an effective PaaS solution.

Kube is very clearly in the expansion phase of its lifecycle but I doubt it will stay this way over the long run.

Additionally, I think the fact that the overall categories are pretty well chosen is a sign of how decently understood the macro problems are and that there's a good reason to hope that you'll end in a place with 1-2 dominant players in each.