Into the Borg – SSRF inside Google production network • Hacker News Board

Re: Into the Borg – SSRF inside Google production network

#61

Post by davidopp__ » Sat, Jul 21, 2018, 7:15 PM UTC

jsmthrowaway wrote: post #53

colohan wrote: post #52

Earlier quoted context omitted.

In this thread there is a repeated meme of "Borg is way more scalable than Kubernetes, and will always be so". But this ignores a lot of the history of Borg. When Borg was first created, it was not nearly as scalable as its current incarnation. We hit scalability bugs and limitations all the time! (I was working on a team which was exploring the scalability limits of MapReduce, which was often very good at finding th…

Borg will remain orders of magnitude beyond Kubernetes until Kubernetes is completely rearchitected. It’s not scalability bugs. It’s decisions regarding how the cluster maintains state that hamstring it, and that’s so fundamental to everything it’s not a find/squish loop. As I said in my comment, those major customers (one personal experience, three anecdotally, eight or nine I’ve consulted with) have quietly ruled o…

> Borg will remain orders of magnitude beyond Kubernetes until Kubernetes is completely rearchitected. It’s not scalability bugs. It’s decisions regarding how the cluster maintains state that hamstring it, and that’s so fundamental to everything it’s not a find/squish loop.

Can you say more about this? Borgmaster uses Paxos for replicating checkpoint data, and etcd uses Raft for replicating the equivalent data, but these are really just two flavors of the same algorithm. I don't doubt that there are probably more efficient ways that Kubernetes could handle state (I don't claim to be an expert in that area), but I don't think they're approaches that would look any more like Borg than Kubernetes does.

If you're at liberty to do so, could you say what orchestrators the customers you mentioned chose in lieu of Kubernetes? What scale are they running at for a single cluster?

[Disclaimer: I work on Kubernetes/GKE at Google.]

Re: Into the Borg – SSRF inside Google production network

#62

Post by jbeda » Sat, Jul 21, 2018, 9:31 PM UTC

GauntletWizard wrote: post #16

puzzle wrote: post #12

> Google is still relying on Borg for its internal production infrastructure, but I can tell you it’s not because of the design of Borg interfaces! No matter how spartan, the Borg status pages are more helpful than most Kubernetes UIs out there when it comes to debugging a problem in depth, i.e. past CPU and memory graphs. Part of that is made possible by applications exposing debugging endpoints and telling Borg abo…

Yeah, I would kill for the k8s pod contract to have something like the borg status line.

It's something we've talked about on and off. Never seems to make it happen.

Re: Into the Borg – SSRF inside Google production network

#63

Post by jmillikin » Sun, Jul 22, 2018, 12:10 AM UTC

jsmthrowaway wrote: post #49

throwawaykbfud wrote: post #47

Earlier quoted context omitted.

There's a lot of FUD in here, like suggesting that Google wouldn't use something written in Go for this purpose (lol,) suggesting an open source platform that can span multiple clouds is an attempt to lock people into GKE (lol 2x,) and suggesting that Kubernetes is "married" to Docker (CNI? CRI?) or isn't extensible (the entire gRPC/REST API? custom resources? device plugins?) People use Kubernetes for the ecosystem,…

> I don't disagree that Kubernetes is not positioned to be a replacement for Borg Good. Because that was my point, but the verbiage “don’t disagree” says a lot. We agree, except for the timeline. We will all be dead before Borg is replaced with Kubernetes. You can take that to the bank. Kinda weird to fire up a throwaway, presumably to conceal your Google credentials, then attack a Xoogler who used to work on Borg SR…

  > Kinda weird to fire up a throwaway, presumably to conceal
  > your Google credentials, then attack a Xoogler who used to
  > work on Borg SRE (alongside sjh under Peter Dahl, and long
  > enough my NDA has long since lapsed)

That's an interesting way to phrase it. I also worked on Borg SRE with Seth under Peter (for six years), and I don't remember anyone else with the same initials as me being present during that time. Just in case I was forgetting someone, I checked the Borg paper (https://storage.googleapis.com/pub-tools-public-publication-...) for the SRE credits section -- I remember all of them, but none had your ... personality.

Luckily, the very first hit for a search of [[ jsmthrowaway site:news.ycombinator.com ]] is your old comment from 2013 (https://news.ycombinator.com/item?id=6750805), in which you discuss being terminated after two months due to failing a background check. That explains why I couldn't remember you, but it doesn't explain why you think two months of reading "Borg 101" tutorials has given you meaningful insight into which parts of the Kubernetes implementation are difficult to scale.

Re: Into the Borg – SSRF inside Google production network

#64

Post by jsmthrowaway » Sun, Jul 22, 2018, 8:52 AM UTC

jmillikin wrote: post #63

jsmthrowaway wrote: post #49

Earlier quoted context omitted.

> I don't disagree that Kubernetes is not positioned to be a replacement for Borg Good. Because that was my point, but the verbiage “don’t disagree” says a lot. We agree, except for the timeline. We will all be dead before Borg is replaced with Kubernetes. You can take that to the bank. Kinda weird to fire up a throwaway, presumably to conceal your Google credentials, then attack a Xoogler who used to work on Borg SR…

> Kinda weird to fire up a throwaway, presumably to conceal > your Google credentials, then attack a Xoogler who used to > work on Borg SRE (alongside sjh under Peter Dahl, and long > enough my NDA has long since lapsed) That's an interesting way to phrase it. I also worked on Borg SRE with Seth under Peter (for six years), and I don't remember anyone else with the same initials as me being present during that time.…

I love when a thread has been dead for hours and the roaches come out of the woodwork to begin turning a conversation personal because it was linked somewhere, and I obviously and emphatically hit a nerve by criticizing something you hold dear. That’s the only reason I’m still refreshing it at this point.

Looking me up (Lord have mercy) and swiping at me as you have here tells me volumes more about you than the effect you intended. What are you expecting? It really says something about Valley culture that I take a swing at Kubernetes, of all bloody things, not gender equality, not politics, not the worth of your contributions to society such as they are, cluster orchestration software, and the absolute worst of you comes out. No factual content, just your smug satisfaction after trying to take an Internet username down a peg. You made the world a better place today. If I cared about you, I might find the whole thing rather sad. I suspect you already do, or will soon, so I need not expend the effort.

Don’t worry, I remember you. If you think you have a handle on my personality from reading disdain I vent on Hacker News, you’re much dumber than I remember, which is disappointing given the respect I have for what little I know of your work. Shame, really.

Re: Into the Borg – SSRF inside Google production network

#65

Post by justicezyx » Sun, Jul 22, 2018, 11:37 AM UTC

repolfx wrote: post #56

davidopp__ wrote: post #41

Earlier quoted context omitted.

[Disclaimers: I worked on Borg and Omega, and currently work on Kubernetes/GKE. Everything here is my personal opinion.] There's a lot to unpack here, but I'll do my best. I don't see Kubernetes locking people into GKE. There's an extensive conformance program ( https://github.com/cncf/k8s-conformance ) administered by the CNCF. AWS and Azure both have certified hosted Kubernetes offerings. Portability is in Google's…

As a Xoogler myself, I have always wondered about the logic of "we can't open source X because it uses too many libraries and is too integrated". The obvious answer is, OK, open source the libraries and refactor the integrations to make them more flexible. Reimplementing all of Borg from scratch seems crazy to me given the huge effort that went into it. Does Google want an open source cluster infrastructure or not? I…

Generally Google software has a bottom up completely different approach to industry norm/standard. And the divergence started from Google’s very beginning.

Open sourcing system software from its internal state requires the same amount of work as rewriting, plus the effort to morph interfaces and internals to fit external needs, plus changes to internal workloads (assuming a unified stack internal & external).

Re: Into the Borg – SSRF inside Google production network

#66

Post by repolfx » Sun, Jul 22, 2018, 11:51 AM UTC

jsnell wrote: post #57

repolfx wrote: post #56

Earlier quoted context omitted.

As a Xoogler myself, I have always wondered about the logic of "we can't open source X because it uses too many libraries and is too integrated". The obvious answer is, OK, open source the libraries and refactor the integrations to make them more flexible. Reimplementing all of Borg from scratch seems crazy to me given the huge effort that went into it. Does Google want an open source cluster infrastructure or not? I…

What's the size of the transitive dependency graph of Borg? 10MLOC? 50MLOC? 100MLOC? I have no idea. But it's a lot of code no matter what. Open sourcing that much code is a huge undertaking, unless you're just planning to throw it over the wall with no expectation of external people working on it. On the other hand starting from scratch you get to grow the community and the codebase in lockstep.

It may be a large undertaking but yes, it's clearly still less work to release code that exists and build a community around it, than rewrite it all from scratch and also build a community around that too.

Re: Into the Borg – SSRF inside Google production network

#67

Post by repolfx » Sun, Jul 22, 2018, 11:59 AM UTC

justicezyx wrote: post #65

repolfx wrote: post #56

Earlier quoted context omitted.

As a Xoogler myself, I have always wondered about the logic of "we can't open source X because it uses too many libraries and is too integrated". The obvious answer is, OK, open source the libraries and refactor the integrations to make them more flexible. Reimplementing all of Borg from scratch seems crazy to me given the huge effort that went into it. Does Google want an open source cluster infrastructure or not? I…

Generally Google software has a bottom up completely different approach to industry norm/standard. And the divergence started from Google’s very beginning. Open sourcing system software from its internal state requires the same amount of work as rewriting, plus the effort to morph interfaces and internals to fit external needs, plus changes to internal workloads (assuming a unified stack internal & external).

I worked on google3 for years, most likely some of the code I wrote is still there. I've also done a lot of open source coding too. I'm quite familiar with the structure of google3 and it's not as different as you claim - Borg is a bunch of C++ libraries and programs that depend on each other, nothing magic about that.

So I completely disagree that open sourcing code is as hard as rewriting it from scratch. I think if you tried to argue that to anyone outside the Google bubble they'd think you were crazy. Writing code is hard work! Uploading it to github and creating some project structures around it is vastly less work.

I can't help wondering if this is engineers looking for new promotion-worthy projects.

Re: Into the Borg – SSRF inside Google production network

#68

Post by jmillikin » Sun, Jul 22, 2018, 3:17 PM UTC

jsmthrowaway wrote: post #64

jmillikin wrote: post #63

Earlier quoted context omitted.

> Kinda weird to fire up a throwaway, presumably to conceal > your Google credentials, then attack a Xoogler who used to > work on Borg SRE (alongside sjh under Peter Dahl, and long > enough my NDA has long since lapsed) That's an interesting way to phrase it. I also worked on Borg SRE with Seth under Peter (for six years), and I don't remember anyone else with the same initials as me being present during that time.…

I love when a thread has been dead for hours and the roaches come out of the woodwork to begin turning a conversation personal because it was linked somewhere, and I obviously and emphatically hit a nerve by criticizing something you hold dear. That’s the only reason I’m still refreshing it at this point. Looking me up (Lord have mercy) and swiping at me as you have here tells me volumes more about you than the effec…

I found this post on the HN front page. I take it somewhat personally because you're using an account with my initials, claiming to be a member of the team I used to work on, and using it to pretend knowledge of a system you used for less time than a typical intern.

https://www.clevescene.com/64-and-counting/archives/2010/12/...

Re: Into the Borg – SSRF inside Google production network

#69

Post by jsmthrowaway » Sun, Jul 22, 2018, 3:27 PM UTC

jmillikin wrote: post #68

jsmthrowaway wrote: post #64

Earlier quoted context omitted.

I love when a thread has been dead for hours and the roaches come out of the woodwork to begin turning a conversation personal because it was linked somewhere, and I obviously and emphatically hit a nerve by criticizing something you hold dear. That’s the only reason I’m still refreshing it at this point. Looking me up (Lord have mercy) and swiping at me as you have here tells me volumes more about you than the effec…

I found this post on the HN front page. I take it somewhat personally because you're using an account with my initials, claiming to be a member of the team I used to work on, and using it to pretend knowledge of a system you used for less time than a typical intern. https://www.clevescene.com/64-and-counting/archives/2010/12/...

Good thing it wasn’t my primary responsibility and I didn’t, you know, study the system Google expected me to administer for a living. That sort of knowledge might have shaped my career if I had spent two months studying a software system.

Oh wait, I did, and it did.

Come on, John, I’ve shipped entire products in less than two months. So have you, I’m sure. You really think I passed a SRE-SWE panel then couldn’t figure out a complex software system in eight weeks? Is this really the approach you’d want someone to remember you for? Butthurt and lashing out because I don’t agree with you on software you’ve convinced Stripe to pay you to fuck around with, and assailing credentials instead of listening to what might be a valuable point, or even disputing it?

I always think about the people who flip me off from the safety of halfway down the exit ramp when I’m in threads like these. Not that I’d do anything about it, because I have priors, but still, I smile.

I don’t have a middle name, by the way.

Re: Into the Borg – SSRF inside Google production network

#70

Post by justicezyx » Mon, Jul 23, 2018, 4:59 AM UTC

repolfx wrote: post #67

justicezyx wrote: post #65

Earlier quoted context omitted.

Generally Google software has a bottom up completely different approach to industry norm/standard. And the divergence started from Google’s very beginning. Open sourcing system software from its internal state requires the same amount of work as rewriting, plus the effort to morph interfaces and internals to fit external needs, plus changes to internal workloads (assuming a unified stack internal & external).

I worked on google3 for years, most likely some of the code I wrote is still there. I've also done a lot of open source coding too. I'm quite familiar with the structure of google3 and it's not as different as you claim - Borg is a bunch of C++ libraries and programs that depend on each other, nothing magic about that. So I completely disagree that open sourcing code is as hard as rewriting it from scratch. I think i…

To rebuttal your statements, I seem need to reveal a lot of technical details. You did not mention what type of software you were open sourcing when you were in Google. But it seems our overlap in knowledge is rather small.

I'll leave this open.

But I want to emphasize that what I stated are reasonable reasons for open sourcing by writing from scratch.