Cloudflare Reverse Proxies Are Dumping Uninitialized Memory • Hacker News Board

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#351

Post by wilde » Fri, Feb 24, 2017, 2:18 AM UTC

pepve wrote: post #318

wilde wrote: post #247

Earlier quoted context omitted.

Are you guys planning to release the list so we can all change our passwords on affected services? Or are you planning on letting those services handle the communication?

That list contains domains where the bug was triggered. The information exposed through the bug though can be from any domain that uses Cloudflare. So: all services that have one or more domains served through Cloudflare may be affected. The consensus seem to be that no one discovered this before now, and no bad guys have been scraping this leak for valuable data (passwords, OAuth tokens, PII, other secrets). But the…

Ah, gotcha. Thanks for explaining!

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#352

Post by roca » Fri, Feb 24, 2017, 2:18 AM UTC

infinity0 wrote: post #307

PuffinBlue wrote: post #187

Earlier quoted context omitted.

> The infosec team worked to identify URIs in search engine caches that had leaked memory and get them purged. With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains. The leaked memory has been purged with the help of the search engines. So I tried it too, and there's still data cached there. Am…

Could someone enlighten me on why malloc and free don't automatically zero memory by default? Someone pointed me to MALLOC_PERTURB_ and I've just run a few test programs with it set - including a stage1 GCC compile, which granted may not be the best test - and it really doesn't dent performance by much. (edit: noticeably, at all , in fact) People who prefer extreme performance over prudent security should be the ones…

Zeroing on alloc/free probably wouldn't have helped much with this bug. Data in live allocations would still be leaked.

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#353

Post by dantiberian » Fri, Feb 24, 2017, 2:19 AM UTC

jgrahamc wrote: post #222

dantiberian wrote: post #163

From Twitter: "@taviso their post-mortem indicates this would've been exploitable only 4 days prior to your initial contact. Is that info invalid?" - https://twitter.com/pmoust/status/834916647873961984 "@pmoust Yes, they worded it confusingly. It was exploitable for months, we have the cached data." - https://twitter.com/taviso/status/834918182640996353

From my blog on this: The three features implicated were rolled out as follows. The earliest date memory could have leaked is 2016-09-22. 2016-09-22 Automatic HTTP Rewrites enabled 2017-01-30 Server-Side Excludes migrated to new parser 2017-02-13 Email Obfuscation partially migrated to new parser 2017-02-18 Google reports problem to Cloudflare and leak is stopped

With respect, the blog post buries the user with details. In my opinion, there should have been in bold at the top something like:

Title: Security report on memory disclosure caused by Cloudflare parser bug

(This is a security report, "incident" underplays this. Memory leak sounds a lot more innocuous than memory disclosure).

Data from any website that was proxied via Cloudflare since September 22, 2016 may have been leaked to third parties via a bug in Cloudflare's HTML parser. Operators using Cloudflare should:

* Invalidate session cookies

* Reset user passwords

* Rotate secrets

* Inform users that private data (chats, pictures, passwords, ...) may have been inadvertently leaked by Cloudflare.

* ...

Users using websites proxied by Cloudflare should:

* Reset their passwords

* Log in/out of sessions to remove session tokens

(Begin rest of post)

Last Friday, Tavis Ormandy from Google’s Project Zero contacted Cloudflare to report a security problem with our edge servers. He was seeing corrupted web pages being returned by some HTTP requests run through Cloudflare. ...

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#354

Post by nikisweeting » Fri, Feb 24, 2017, 2:19 AM UTC

curuinor wrote: post #264

Can we start a list of affected right now? I found: OKCupid Uber people claiming 1Password, can't find Reddit Lyft Yelp Pingdom Digital Ocean Montecito Bank and Trust

I'm compiling a list of affected domains here, please submit PRs: https://github.com/pirate/sites-using-cloudflare

I'm currently running a DNS scraper to find more.

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#355

Post by djKianoosh » Fri, Feb 24, 2017, 2:21 AM UTC

espadrine wrote: post #125

tptacek wrote: post #4

Oh, my god. Read the whole event log. If you were behind Cloudflare and it was proxying sensitive data (the contents of HTTP POSTs, &c), they've potentially been spraying it into caches all across the Internet; it was so bad that Tavis found it by accident just looking through Google search results. The crazy thing here is that the Project Zero people were joking last night about a disclosure that was going to keep e…

It is far from over, too! Google Cache still has loads of sensitive information, a link away! Look at this, click on the downward arrow, "Cached": https://www.google.com/search?q="CF-Host-Origin-IP:"+"author... (And then, in Google Cache, "view source", search for "authorization".) (Various combinations of HTTP headers to search for yield more results.)

I searched for

"CF-Host-Origin-IP:" token

.... uhm is that what I think I'm seeing???

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#356

Post by nikisweeting » Fri, Feb 24, 2017, 2:21 AM UTC

no_protocol wrote: post #45

Is there a list of sites potentially affected? I'm assuming I need to change my passwords on a significant number of sites. So far none of them have alerted me to a potential breach. Would love to have a head start.

Compiling a list here: https://github.com/pirate/sites-using-cloudflare

It will be updated as we find more.

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#357

Post by nine_k » Fri, Feb 24, 2017, 2:21 AM UTC

dkarapetyan wrote: post #75

Every piece of dependency in your stack is a vulnerability vector. I feel like this is the only sane assumption to make these days. Yesterday I was thinking of doing some stuff with cloudflare and today I'm reading this report. The modern web requires a paranoid attitude.

Only the paranoid survive, Andy Groves famously said.

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#358

Post by snaky » Fri, Feb 24, 2017, 2:22 AM UTC

Kalium wrote: post #286

kevcampb wrote: post #252

Earlier quoted context omitted.

In other words, Baidu are currently sitting on a treasure trove of keys and passwords.

Possibly not, Baidu and CloudFlare have a well-documented long-term partnership.

Maybe there's much more to worry about Baidu's particular not-so-well documented, but longer-term partnerships.

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#359

Post by nikisweeting » Fri, Feb 24, 2017, 2:22 AM UTC

kmfrk wrote: post #106

So how does one find or generate a list of companies using CloudFlare to figure out how you're affected - kinda like HaveIBeenPwned.com?

While theoretically many non-cloudflare sites can be affected, I'm compiling a list of known cloudflare-using domains here: https://github.com/pirate/sites-using-cloudflare

It can serve as a starting point, just Ctrl+F for domains you use.

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#360

Post by amalcon » Fri, Feb 24, 2017, 2:22 AM UTC

infinity0 wrote: post #307

PuffinBlue wrote: post #187

Earlier quoted context omitted.

> The infosec team worked to identify URIs in search engine caches that had leaked memory and get them purged. With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains. The leaked memory has been purged with the help of the search engines. So I tried it too, and there's still data cached there. Am…

Could someone enlighten me on why malloc and free don't automatically zero memory by default? Someone pointed me to MALLOC_PERTURB_ and I've just run a few test programs with it set - including a stage1 GCC compile, which granted may not be the best test - and it really doesn't dent performance by much. (edit: noticeably, at all , in fact) People who prefer extreme performance over prudent security should be the ones…

Some old IBM environments initialized fresh allocations to 0xDEADBEEF, which had the advantage that the result you got from using such memory would (usually) be obviously incorrect. The fact that it was done decades ago is pretty good evidence that it's not about the actual initialization cost: these things cost a lot more back then.

What changed is the paged memory model: modern systems don't actually tie an address to a page of physical RAM until the first time you try to use it (or something else on that page). Initializing the memory on malloc() would "waste" memory in some cases, where the allocation spans multiple pages and you don't end up using the whole thing. Some software assumes this, and would use quite a bit of extra RAM if malloc() automatically wiped memory. It would also tend to chew through your CPU cache, which mattered less in the past because any nontrivial operation already did that.

I personally don't think this is a good enough reason, but it is a little more than just a minor performance issue.

That all being said, while it would likely have helped slightly in this case, it would not solve the problem: active allocations would still be revealed.