Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Into the Borg – SSRF inside Google production network

opnsec.com

Post reply on HN
Print view View original

11–20 of 79 posts

Re: Into the Borg – SSRF inside Google production network

#11

Post by beering »

powera wrote: post #8
elvinyung wrote: post #4

> I should mention that Borg, like Kubernetes, relies on containers like Docker Hmm, containers _like_ Docker, or Docker? I thought Google used lmctfy since long before Docker?

cgroups were written at Google, and have been used internally for a very long time; they provide "container"-like limits on resource usage for a group of processes. I assume that Google isn't using Docker internally for production services, but don't know for sure (and I assume anyone who does know for sure can't tell you).

It also just doesn't make sense for Google to use Docker (or even Kubernetes) for their core infra. They've been the foremost leader in distributing containerized applications across data centers. Whatever they've already built is almost certainly more battle-tested and more customized to their needs than anything public that's based on their concepts.

Re: Into the Borg – SSRF inside Google production network

#12

Post by puzzle »

> Google is still relying on Borg for its internal production infrastructure, but I can tell you it’s not because of the design of Borg interfaces!

No matter how spartan, the Borg status pages are more helpful than most Kubernetes UIs out there when it comes to debugging a problem in depth, i.e. past CPU and memory graphs. Part of that is made possible by applications exposing debugging endpoints and telling Borg about them.

Re: Into the Borg – SSRF inside Google production network

#13

Post by elvinyung »

beering wrote: post #11
powera wrote: post #8

Earlier quoted context omitted.

cgroups were written at Google, and have been used internally for a very long time; they provide "container"-like limits on resource usage for a group of processes. I assume that Google isn't using Docker internally for production services, but don't know for sure (and I assume anyone who does know for sure can't tell you).

It also just doesn't make sense for Google to use Docker (or even Kubernetes) for their core infra. They've been the foremost leader in distributing containerized applications across data centers. Whatever they've already built is almost certainly more battle-tested and more customized to their needs than anything public that's based on their concepts.

Sure it's battle tested, but it's quite possible that the advantage of having a software with more eyeballs on it is even more valuable.

Last I heard, the biggest reason that Google still uses Borg instead of Kubernetes is mostly because of switching costs.

Re: Into the Borg – SSRF inside Google production network

#14

Post by outworlder »

tptacek wrote: post #3

This is a great find. SSRF is a really unappreciated vulnerability; it is usually game-over. That it came from a Caja audit adds some tasty irony. A friendly word of advice: when you find flaws like this (you, the reader, not you, the guy who wrote this post), think carefully before disclosing internal network details you discover like this writer did. The internal details of a target network don't become public doma…

I'm still amazed that this individual published as much data as he did. I don't see where it says they have permission to do so.

Specially because:

> I hope they won’t beat me with a stick for disclosing any of this

This tells me that this wasn't cleared with them. Doesn't sound like a smart move. If your gut is telling you it may be a bad idea, it could be because it is...

Re: Into the Borg – SSRF inside Google production network

#15

Post by bowmessage »

outworlder wrote: post #14
tptacek wrote: post #3

This is a great find. SSRF is a really unappreciated vulnerability; it is usually game-over. That it came from a Caja audit adds some tasty irony. A friendly word of advice: when you find flaws like this (you, the reader, not you, the guy who wrote this post), think carefully before disclosing internal network details you discover like this writer did. The internal details of a target network don't become public doma…

I'm still amazed that this individual published as much data as he did. I don't see where it says they have permission to do so. Specially because: > I hope they won’t beat me with a stick for disclosing any of this This tells me that this wasn't cleared with them. Doesn't sound like a smart move. If your gut is telling you it may be a bad idea, it could be because it is...

Yeah, while it might be interesting to know what Gmail's system user is, it's not really related at all to the vuln. I wouldn't be surprised if payment of the bounty was contingent on some kind of ToS acceptance about responsible disclosure...

Re: Into the Borg – SSRF inside Google production network

#16

Post by GauntletWizard »

puzzle wrote: post #12

> Google is still relying on Borg for its internal production infrastructure, but I can tell you it’s not because of the design of Borg interfaces! No matter how spartan, the Borg status pages are more helpful than most Kubernetes UIs out there when it comes to debugging a problem in depth, i.e. past CPU and memory graphs. Part of that is made possible by applications exposing debugging endpoints and telling Borg abo…

Yeah, I would kill for the k8s pod contract to have something like the borg status line.

Re: Into the Borg – SSRF inside Google production network

#17

Post by paulddraper »

tl;dr SSRF is server side request forgery, where you can gain access to private resources by convincing privledged servers to make requests for you.

If you are using network access for security, either don't, it blacklist private IPs (and use public DNS) for untrusted URLs.

Re: Into the Borg – SSRF inside Google production network

#19

Post by kerng »

bpicolo wrote: post #7
luhn wrote: post #6

Not quite relevant to the article: Since it seems to be tricky to properly sanitize URLs for SSRF, I had an idea for safely calling user-defined URLs: Set up an unprivileged non-VPC Lambda function that calls a URL and call all user-defined URLs through the Lambda function. I think it should be bulletproof, anything I'm overlooking?

GCP does it by requiring a header on requests to metadata systems. Require a header on your internal services and make sure you never send that header with user-requested URLs and you can guarantee safety there.

Still allows tons of attacks and recon, like port scans, or even crashes of processes- since internal things might not go through the same fuzzing scrutiny as external endpoints.

Re: Into the Borg – SSRF inside Google production network

#20

Post by threeseed »

puzzle wrote: post #12

> Google is still relying on Borg for its internal production infrastructure, but I can tell you it’s not because of the design of Borg interfaces! No matter how spartan, the Borg status pages are more helpful than most Kubernetes UIs out there when it comes to debugging a problem in depth, i.e. past CPU and memory graphs. Part of that is made possible by applications exposing debugging endpoints and telling Borg abo…

Can't you just use CAdvisor and route metrics to Prometheus/Grafana ?
Post reply on HN