Live data from Hacker News

Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

bugs.chromium.org

751–760 of 1001 posts

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#751
post #749

Earlier quoted context omitted.

>I'm troubled that they went ahead with disclosure before Google crawl team could complete the refresh of their own cache. It sounded like they (cf) were under a lot of pressure to disclose ASAP from project zero and their 7 day requirement...

eastdakota is one of the cloudflare guys, so "they" in that sentence can only refer to Google (see also the previous paragraph/sentences, where eastdakota used "we" for cloudflare).

He's the CEO

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#752
post #248

This sounds to me like an object lesson in "Why you shouldn't write your own HTML parser." Every time I see a dev trying to parse HTML with a custom solution or regex or anything other than a proven OSS library designed to parse HTML I recoil reflexively. Sure, maybe you don't need a parser to see if that strong tag is properly closed but the alternative is ...

You're right in 99+% of cases. But I suspect that the needs of cloudflare for this use case aren't typical of what's expected of an html parser. I'm not certain that there isn't an existing parser that would work for them, but I'm equally not certain that there is.

Even so, if the parser handles security or human safety then it shouldn't be written in C, or even using a parser generator that generates C.

Just use ML, or Rust, or bloody JavaScript for all I care. I don't care if they add a ton to response time, or add 100% perf overhead costs for running the thing.

Having an OS, ssl library, web server etc written in C is bad enough but at least that code has many eyes on it. Companies shouldnt throw their custom made tyres on top of that fire.

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#753

Author of Ragel here. An experienced Ragel programmer would know that when you start setting the EOF pointer you are enabling code paths that never executed before. Like, potentially buggy ones. Eek!

I'm not sure if you've had a chance to look at the Cloudflare blog post yet (https://blog.cloudflare.com/incident-report-on-memory-leak-c...), but while they take full responsibility they do point out (under root cause of the bug) that a generated equality check could be >= instead which would avoid the bug. Obviously GIGO applies and it's their bug, but it might be worth seeing if there's anything you can do on Ragel side?

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#754
In addition to comments here calling the words 'memory leak' disingenuous because it's technically correct but underplays the problem, I'm now seeing articles in non-technical publications referring to the incident as a "leak".

In the wider world the word "leak" doesn't mean memory access patterns, it means deliberate sabotage.

The headline in "The Verge" is "Password and dating site messages leaked by internet giant Cloudflare". That's technically correct too, but also gives completely the wrong message.

Simpler, proactive messaging from Cloudfront might have helped here.

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#755
post #704

This comes around to me as something that just shouldn't have happened. CloudFlare are pretty big on Go, as far as I can tell (and I guess Lua for scripting nginx). Why was this parsing package written in a non memory-safe language? Parsing is one of those "obvious" things easy to mess up; the likelihood of a custom, hand written parser being buggy is pretty high. If it's somehow understood that your library is likel…

This could easily happen in Go as well. All that would be needed is to reuse the buffer in between requests, and rely on the buffer length instead of clearing it. To make it safer you would need to deallocate and reallocate the buffer for each request, but that might be slow. Doing that would fix it for Go, or for C, it would be the same either way. So I'm not convinced that using Go would have helped here.

It's a good point, but at least with Go the leak would be limited to the allocated buffer. This is probably a case where Rust or C++ might be more helpful. Presumably you wouldn't want to allocate a new (variable sized) buffer each time (particularly in a GC language), but you could create a new (bounds checked) slice[1] / array_view[2] / gsl::span / RandomAccessSection[3] each time.

[1] https://doc.rust-lang.org/nightly/std/slice/

[2] https://github.com/rhysd/array_view

[3] https://github.com/duneroadrunner/SaferCPlusPlus#txscoperand...

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#756
post #702

Earlier quoted context omitted.

I won't take the initiative of changing passwords, and I will only be doing it for services that ask me to do it. In my opinion, if my accounts get compromised because the provider uses Cloudflare and leaks my data all over, it's their fault, not mine... It's not my job to guess which services are using Cloudflare, which ones were affected... and further, if my account gets compromised, others presumably will. (PS: O…

The damage is still yours even if its not your fault

I second this, and you'll be even more furious if someone used that to compromise your data/accounts.

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#757
post #701
post #549

Earlier quoted context omitted.

Yea but... seems like a quick run of valgrind would have caught this

Not necessarily. If they just keep reusing a buffer and forget to clear it in between requests there is nothing automated that would find it. Bounds checking languages would not help either - they would only work if they delete and reallocate the buffer on each request, since that's slow it's unlikely anyone would do that. They probably wouldn't even clear the buffer, instead they rely on keeping track of the length…

If they used asan/msan and its support for manually marking regions of memory as invalid/uninitialized, that could have caught such cases too.

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#758

Author of Ragel here. An experienced Ragel programmer would know that when you start setting the EOF pointer you are enabling code paths that never executed before. Like, potentially buggy ones. Eek!

I'm not sure if you've had a chance to look at the Cloudflare blog post yet ( https://blog.cloudflare.com/incident-report-on-memory-leak-c... ), but while they take full responsibility they do point out (under root cause of the bug) that a generated equality check could be >= instead which would avoid the bug. Obviously GIGO applies and it's their bug, but it might be worth seeing if there's anything you can do on Ra…

Well doing that would mean ragel would incorrectly read one character, rather than run off forever. Personally I'd rather have the latter. Much easier to catch with memory checkers. Eventually you try to read some thing you're not allowed to read, or blow something else up, instead of just read the first byte of the int following the buffer, or whatever.

There would have to be an additional bounds check when issuing a goto in an error action, but doing that is contrary to the simple execution model that ragel users have come to rely on.

Gotta ask the question, where was the testing when they altered 7 year old code without the involvement of the original developer?

Re: Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

#760
post #4

Oh, my god. Read the whole event log. If you were behind Cloudflare and it was proxying sensitive data (the contents of HTTP POSTs, &c), they've potentially been spraying it into caches all across the Internet; it was so bad that Tavis found it by accident just looking through Google search results. The crazy thing here is that the Project Zero people were joking last night about a disclosure that was going to keep e…

I would say the crazy thing is a mere t-shirt as their "bug bounty" top tier award given how they've pitched themselves as an extremely secure service. https://hackerone.com/cloudflare I'm sorry but when the reward for breaking into you is basically a massive pinata of personal information...that simply is a bad joke. Security flaws are going to happen and if you aren't going to even offer a reasonable financial rewa…

What would make sense (to me, not a business/marketing guy, nor a lawyer, at all) would be a t-shirt and free subscription as the offered thing, something which costs the company nothing.

Then for anything like this, give publically a bonus gift which makes it worth people reporting to them and not blackmarket selling it. Once it's gone through the legal dept. and so on.

Then they can be very quick with handing out tshirts and so on to any and every microissue report, without the people running triage having to care about amounts or tax or whatever.

Having any kind of publically offered payment for service (beyond a tshirt bounty or services in kind) is just begging for legal issues, right?

Post reply on HN