How we got read access on Google’s production servers

#1

Post by detectify » Fri, Apr 11, 2014, 9:23 AM UTC

tl;dr: How we got $10.000. We were able to upload an XML file to Google. The XML parser was vulnerable to XXE. We got full read access to their production servers, including the /etc/passwd.

How we got read access on Google’s production servers
blog.detectify.com

Re: How we got read access on Google’s production servers

#2

Post by numair » Fri, Apr 11, 2014, 9:45 AM UTC

... And this is why you want to discontinue products and services your engineers can't be motivated to maintain. Amazing.

This should scare anyone who has ever left an old side project running; I could see a lot of companies doing a product/service portfolio review based on this as a case study.

Re: How we got read access on Google’s production servers

#3

Post by NicoJuicy » Fri, Apr 11, 2014, 9:48 AM UTC

A job well done. This is actually impressive and quite interesting to see after what you are searching for (afterwards it seems logical :))

Re: How we got read access on Google’s production servers

#4

Post by cheald » Fri, Apr 11, 2014, 9:55 AM UTC

XML legitimately scares me. The number of scary, twisted things it can do make me shudder every time I write code to parse some XML from anywhere - it just feels like a giant timebomb waiting to happen.

Re: How we got read access on Google’s production servers

#5

Post by raesene3 » Fri, Apr 11, 2014, 9:58 AM UTC

Interesting to see this hit big companies like google. The problem, I think, stems from the idea that most people treat XML parsers as a "black box" and don't enquire too closely as to all the functionality that they support.

Reading the spec. which led to the implementations, can often reveal interesting things, like support for external entities..

Re: How we got read access on Google’s production servers

#6

Post by vidarh » Fri, Apr 11, 2014, 10:07 AM UTC

raesene3 wrote: post #5

Interesting to see this hit big companies like google. The problem, I think, stems from the idea that most people treat XML parsers as a "black box" and don't enquire too closely as to all the functionality that they support. Reading the spec. which led to the implementations, can often reveal interesting things, like support for external entities..

Also horrible defaults in XML parsers. That any XML parsers allow retrieval of DTD's without explicit options specifying allowed sources etc. is beyond me. It's not just local file access, which becomes a security hole when you let users pass you XML files, though that is one of the worst ones.

But the number of times I've seen production apps that turn out to behind the scenes request DTD's or schemas from remote servers regularly have made that one of the first thing I check if I am tasked to maintain or look into anything that parses XML. Often these apps stop working or slow down for seemingly no reason because the DTD or schema becomes unavailable, and nobody understands why.

Re: How we got read access on Google’s production servers

#7

Post by twic » Fri, Apr 11, 2014, 10:08 AM UTC

raesene3 wrote: post #5

Interesting to see this hit big companies like google. The problem, I think, stems from the idea that most people treat XML parsers as a "black box" and don't enquire too closely as to all the functionality that they support. Reading the spec. which led to the implementations, can often reveal interesting things, like support for external entities..

I would say the flaw is that XML parsers will try to resolve external entities on their own, by resolving file paths or whatever. They shouldn't do this by default: they should instead take a programmer-supplied entity resolver and call into that.

They could also provide a canned resolver which hits the local filesystem and/or the web, which programmers could supply if they wanted, but this should not be a default. The programmer should have to explicitly specify that access.

I've had related problems where XML parsers would try to go off and fetch DTDs from the web, then fail, because they were running on firewalled machines that couldn't see the servers hosting the DTDs. That took us by surprise. We installed an entity resolver that looked in a local cache of DTDs instead, which was fairly easy. But i would prefer not to have been surprised.

Also, all this stuff should be running in a jail where it can't even see any interesting files, of course.

Re: How we got read access on Google’s production servers

#8

Post by antocv » Fri, Apr 11, 2014, 10:16 AM UTC

So, when you have read access to googles prod servers, what else would be fun to do besides reading /etc/passwd ?

Getting the source?

Re: How we got read access on Google’s production servers

#9

Post by mixmax » Fri, Apr 11, 2014, 10:18 AM UTC

In large production environments it's almost impossible to avoid bugs - and some of them are going to be nasty. What sets great and security conscious companies apart from the rest is how they deal with them.

This is an examplary response from google. They respond promptly (with humor no less) and thank the guys that found the bug. Then they proceeded to pay out a bounty of $10.000.

Well done google.

Re: How we got read access on Google’s production servers

#10

Post by spindritf » Fri, Apr 11, 2014, 10:30 AM UTC

numair wrote: post #2

... And this is why you want to discontinue products and services your engineers can't be motivated to maintain. Amazing. This should scare anyone who has ever left an old side project running; I could see a lot of companies doing a product/service portfolio review based on this as a case study.

Or just move it to some cheap VPS where it cannot damage other services or your infrastructure.