Into the Borg – SSRF inside Google production network

I’m talking about competitors to GKE, which is the entire thesis of that point and that you refuse to acknowledge, which is why you’re knee jerking FUD. I’m a solo developer. I make a Kubernetes app. Do I think Kubernetes is great if I have to build a CloudFormation thing to spin up all of its infrastructure and run it myself, or would I rather pay big company to do it?

The latter.

GKE. EKS/etc. They exist for that use case because that’s where the lock in happens. Now you’re a Kubernetes deployable and too dug in to get off. For a while, your only option was GKE or (God help you, at first) running it yourself. Now other vendors have stepped up to compete, which is the exact situation Google wanted. They shook out the Kubernetes issues with the early adopters with ops experience, then went after pure developers, and had first mover advantage on that customer until the others were forced to respond. I wish I could tell you how many teams I’ve consulted with who when I say “where are we deploying?” immediately say Kubernetes on GKE without a second thought. That is what Google wanted.

This doesn’t take a rocket scientist to figure out, but if you work on Kubernetes I wouldn’t expect you to know this strategy firsthand. It would come from Diane or her people. You’re just having a hootin’ and hollerin’ grand old time building some open source software with a foundation (ooo! So OpenStack!) and mindshare and all the cool kids surrendering their startup to go all in on Kubernetes. Meanwhile, you’re supported by Google, and in fact were conceptualized, to drive sales to Google Cloud. Sorry.

Did you ever ask yourself what the higher level point of Kubernetes is? Like, why throw an entire team on shipping an entire new infrastructure category just to open source it? To make operations better for the industry? Google competes on operations. To use public company capital to chase a hobby? Think about the leadership decision to initiate Kubernetes, and why it was made.

> but you said a whole lot of other crap that is very much misleading in my opinion

The entire comment was quite clearly in support of Google’s decision calculus to not replace Borg. Your rebuttals are honestly more misleading, in my opinion, than my points, because you’re personally wrapped up in it and that’s coming across.

Borg will remain orders of magnitude beyond Kubernetes until Kubernetes is completely rearchitected. It’s not scalability bugs. It’s decisions regarding how the cluster maintains state that hamstring it, and that’s so fundamental to everything it’s not a find/squish loop.

As I said in my comment, those major customers (one personal experience, three anecdotally, eight or nine I’ve consulted with) have quietly ruled out Kubernetes, either by trying it or prying it apart and deciding not to try it. That feedback isn’t coming. At Borg scale, Kubernetes is very much considered a nonstarter.

I probably don't - my experience is very one-sided, as I've never tried Kubernetes.

What's the size of the transitive dependency graph of Borg? 10MLOC? 50MLOC? 100MLOC? I have no idea. But it's a lot of code no matter what. Open sourcing that much code is a huge undertaking, unless you're just planning to throw it over the wall with no expectation of external people working on it.

On the other hand starting from scratch you get to grow the community and the codebase in lockstep.

Indeed. The universe where you aren’t trying to replace Borg, and instead have other motives for conceiving of Kubernetes in the first place. That’s all I’m trying to say.

Hadoop is nowhere close to the original thinking behind MapReduce, and Kubernetes is nowhere close to the original thinking behind Borg. The difference is Yahoo! wrote one, and the other leads us to speculate about why they’d want to offer such a good concept to the world, then ship something that so spectacularly misses that mark, particularly scalewise, but get away with it because it’s Google doing it.

There are only a couple explanations, especially when you factor in that corporations always act in their interest. That’s not a moral dimension, that’s just how it works. What upside is there for Google having everyone do cluster orchestration the way they do? Really, I challenge you to think of one. When I was there, you couldn’t even say the word Borg externally (despite the original article not being the first security-based leak of it). Operations and engineering velocity was, and remains, a crucial competitive advantage for Google. Since I’ve left, there’s a clear shift toward teaching the industry how to do operations, from the Borg paper on through to the SRE book. Why? Why suddenly cede that competitive advantage, the ability to ship a whole production service in like, a weekend?

Because you do it poorly, drive people to Google Cloud because administering Kubernetes at any scale is like self-flagellating with a rake (don’t get me started and be satisfied that I’m speaking from ulcers here), and keep ahead of the competition who are forced to respond to GKE now that a whole community is clamoring for Docker in prod. It’s the only explanation that fits. Maybe it wasn’t even intentional, but it’s what has played out.

Another possible explanation is that Google wanted to end run Mesos, which was threatening to dominate until Kubernetes got a bunch of mindshare and venture capital. Mesos, with care and careful choice of schedulers, can be on the same field as Borg. Twitter is very happy with it at scales that would decimate Kubernetes, as well as many others I cannot name. Google’s VC endeavors have been dumping into Kubernetes startups, and now you don’t hear about Mesos any more. Just like Docker before it, we just can’t get enough bad operational systems as a profession, apparently.

My whole career I’ve been obsessed with high scale orchestration because it is a uniquely difficult problem. I saw Borg and went Jesus, we are all behind the curve as an industry. Then the first few releases of Kubernetes came out and I killed it dead with a few dozen nodes and moderate load. It’s gotten better, yes, but Borg has literally centuries of engineering in it, and a steering committee is going to push Kubernetes to OpenStack or Java status (I’d argue it’s already there, given that people are now shipping distributions of it). And now operations as a discipline suffers for it. I can’t get a SRE gig any more without a dev team who doesn’t need Kubernetes, at all, making it a showstopper to ship.

Kinda funny, by the way, that this thread has been off the front page for hours and is still active with new faces. I must have ruffled eng-misc. :)