Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Into the Borg – SSRF inside Google production network

opnsec.com

Post reply on HN
Print view View original

21–30 of 79 posts

Re: Into the Borg – SSRF inside Google production network

#21

Post by puzzle »

threeseed wrote: post #20
puzzle wrote: post #12

> Google is still relying on Borg for its internal production infrastructure, but I can tell you it’s not because of the design of Borg interfaces! No matter how spartan, the Borg status pages are more helpful than most Kubernetes UIs out there when it comes to debugging a problem in depth, i.e. past CPU and memory graphs. Part of that is made possible by applications exposing debugging endpoints and telling Borg abo…

Can't you just use CAdvisor and route metrics to Prometheus/Grafana ?

They're sometimes interactive HTTP endpoints, not metrics, although metrics are served by one of them (/varz, described in the SRE book). They're informally called z pages and are the inspiration behind the /debug handlers that Go's net/trace package installs:

https://pbs.twimg.com/media/CebGW64W4AAJ_sc?format=jpg

OpenCensus has a bunch, too:

https://opencensus.io/zpages/

Re: Into the Borg – SSRF inside Google production network

#22

Post by jamesblonde »

puzzle wrote: post #12

> Google is still relying on Borg for its internal production infrastructure, but I can tell you it’s not because of the design of Borg interfaces! No matter how spartan, the Borg status pages are more helpful than most Kubernetes UIs out there when it comes to debugging a problem in depth, i.e. past CPU and memory graphs. Part of that is made possible by applications exposing debugging endpoints and telling Borg abo…

Yarn apps typically expose a similar http endpoint for app status reports.

Re: Into the Borg – SSRF inside Google production network

#24

Post by dilyevsky »

elvinyung wrote: post #13
beering wrote: post #11

Earlier quoted context omitted.

It also just doesn't make sense for Google to use Docker (or even Kubernetes) for their core infra. They've been the foremost leader in distributing containerized applications across data centers. Whatever they've already built is almost certainly more battle-tested and more customized to their needs than anything public that's based on their concepts.

Sure it's battle tested, but it's quite possible that the advantage of having a software with more eyeballs on it is even more valuable. Last I heard, the biggest reason that Google still uses Borg instead of Kubernetes is mostly because of switching costs.

I don't know where you heard that but it's simply not feasible to run k8s at the scale they run Borg. Not to mention hundreds if not thousands of features it's missing.

Re: Into the Borg – SSRF inside Google production network

#25

Post by puzzle »

jamesblonde wrote: post #22
puzzle wrote: post #12

> Google is still relying on Borg for its internal production infrastructure, but I can tell you it’s not because of the design of Borg interfaces! No matter how spartan, the Borg status pages are more helpful than most Kubernetes UIs out there when it comes to debugging a problem in depth, i.e. past CPU and memory graphs. Part of that is made possible by applications exposing debugging endpoints and telling Borg abo…

Yarn apps typically expose a similar http endpoint for app status reports.

Yeah, but with Borg you can have many of them, all directly accessible with one click from the Borg job status page: e.g. a general status, a view of the work queue if it's something like a crawler, incoming and outgoing RPCs, memory/cpu profiles, etc. You don't want them all on the same page for performance reasons: they would eat up a lot of temporary memory/cycles, hold important locks for long periods of time, etc. And if you need to look at a bunch of containers, you don't want to navigate to subpages for each of them.

Re: Into the Borg – SSRF inside Google production network

#26

Post by londons_explore »

outworlder wrote: post #14
tptacek wrote: post #3

This is a great find. SSRF is a really unappreciated vulnerability; it is usually game-over. That it came from a Caja audit adds some tasty irony. A friendly word of advice: when you find flaws like this (you, the reader, not you, the guy who wrote this post), think carefully before disclosing internal network details you discover like this writer did. The internal details of a target network don't become public doma…

I'm still amazed that this individual published as much data as he did. I don't see where it says they have permission to do so. Specially because: > I hope they won’t beat me with a stick for disclosing any of this This tells me that this wasn't cleared with them. Doesn't sound like a smart move. If your gut is telling you it may be a bad idea, it could be because it is...

The page he accessed probably contained thousands of Google borg jobs. Many about secret or future projects for example. Many giving implementation details for secret sauce algorithms (eg. oh - they precompute all possible misspellings for their spell corrector via this mapreduce!). Simply knowing the fleet wide CPU, network and RAM usage for Gmail would give a competitor a lot of knowledge into the probable running costs of the service for example.

In this case he exposed a tiny proportion of what he accessed.

Sounds fairly reasonable to me.

Re: Into the Borg – SSRF inside Google production network

#28

Post by bogomipz »

powera wrote: post #8
elvinyung wrote: post #4

> I should mention that Borg, like Kubernetes, relies on containers like Docker Hmm, containers _like_ Docker, or Docker? I thought Google used lmctfy since long before Docker?

cgroups were written at Google, and have been used internally for a very long time; they provide "container"-like limits on resource usage for a group of processes. I assume that Google isn't using Docker internally for production services, but don't know for sure (and I assume anyone who does know for sure can't tell you).

I've heard a few times that Google's containers actually run inside of VMs. I'm curious if anyone knows what their VM implementation is or what its based on?

Re: Into the Borg – SSRF inside Google production network

#29

Post by menage »

> it’s not because of the design of Borg interfaces

As the original creator of the Borglet status page, I think it's not really accurate to describe it as "designed". :-)

It was more a case of gradual semi-random evolution, with people (mostly me, at least in the old days - I've no idea how much it might have changed in the last eight years) adding things that seemed like they could help Google engineers trying to track down issues with their Borg jobs. User-friendliness for random SSRF hackers was definitely not high on the priority list.

Re: Into the Borg – SSRF inside Google production network

#30

Post by rrdharan »

bogomipz wrote: post #28
powera wrote: post #8

Earlier quoted context omitted.

cgroups were written at Google, and have been used internally for a very long time; they provide "container"-like limits on resource usage for a group of processes. I assume that Google isn't using Docker internally for production services, but don't know for sure (and I assume anyone who does know for sure can't tell you).

I've heard a few times that Google's containers actually run inside of VMs. I'm curious if anyone knows what their VM implementation is or what its based on?

Mostly upstream KVM with a custom replacement for QEMU:

https://cloudplatform.googleblog.com/2017/01/7-ways-we-harde...

Post reply on HN