Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Details of the Cloudflare outage on July 2, 2019

blog.cloudflare.com

Post reply on HN
Print view View original

91–100 of 159 posts

Re: Details of the Cloudflare outage on July 2, 2019

#91

Post by LukeShu »

stevens32 wrote: post #85

For the regex novices here, would anyone mind explaining what that pattern is meant to match? More specifically, what `. (?:. =.*)` is meant to do?

The appendix does a pretty good job, but the TL;DR is: it will match any string containing 1 or more equal signs.

Slightly more verbosely, it will match [0-or-more bytes of anything] followed by [0-or-more bytes of anything] followed by [an equal sign] followed by [0-or-more bytes of anything]. The expensive part is that it can't decide where the first grouping of [0-or-more bytes of anything] starts and the second grouping begins. It doesn't matter where the division is, of course, but many regex engines use an exponential-time algorithm for that, even though an obvious liner-time algorithm exists (and pre-dates the exponential-time algorithm!).

Re: Details of the Cloudflare outage on July 2, 2019

#92

Post by meowface »

stevens32 wrote: post #85

For the regex novices here, would anyone mind explaining what that pattern is meant to match? More specifically, what `. (?:. =.*)` is meant to do?

The bottom of the post elaborates it.

They said it was for XSS detection. I think the purpose was to identify reflected XSS by looking for paths or headers containing JavaScript-esque variable assignment (JS keywords/syntax preceding "something=something"), but not 100% sure.

Re: Details of the Cloudflare outage on July 2, 2019

#93

Post by f2f »

Twirrim wrote: post #87
f2f wrote: post #81

Earlier quoted context omitted.

RE2 is bullet proof. It is the defacto re engine used by anyone looking to deploy regexes to the wild (used in the now defunct google code search, for example). It has a track record. Russ Cox, its author, was affiliated with Ken Thompson from very early in his career. Rust also has a good pedigree for not being faulty. BurntSushi, the author of rust's regex crate also has a good pedigree... We switched to RE2 for a…

I'm absolutely not denying that RE2 is great. Not in the slightest. I even agree with their idea to switch towards it or the Rust one. Changing anything brings an element of risk, and changing quickly to it, even more so, which is essentially what they're proposing doing. That's where my concern lies. Their current approach clearly has issues, but it has been running in production for several years now and those issu…

i concur with your assessment. i'm sure cloudflare will be cautious and not rush with deployment after the switch. lesson learned?

Re: Details of the Cloudflare outage on July 2, 2019

#94

Post by nullwasamistake »

blr246 wrote: post #5

Appreciate the detail here. It's a great writeup. Wondering what folks think about one of the changes: 5. Changing the SOP to do staged rollouts of rules in the same manner used for other software at Cloudflare while retaining the ability to do emergency global deployment for active attacks. One concern I'd have is whether or not I'm exercising the global rollout procedure often enough to be confident it works when i…

The main problem is that their Regex library doesn't have a recrusion limit. I'm honestly amazed they've been able to scale Lua scripts to the point they can use it as a global WAF. Knowing this, it may be easy to create attacks against their filters.

My takeaway is that it's time to move to a custom solution using a more flexible language. A simple async watchdog on total rule execution time would have prevented this. When running tons of Regex rules I'm amazed they didn't have this

Re: Details of the Cloudflare outage on July 2, 2019

#96

Post by hn_throwaway_99 »

One thing that was interesting to me:

The outage was caused by a regex that ended up doing a lot of backtracking, which caused PCRE, the regex engine, to essentially handle a runaway expression.

This reminded me of a HN post from a couple months back by the author of Google Code Search, and how it worked: https://swtch.com/~rsc/regexp/regexp4.html . Interestingly, he wrote his own regex engine, RE2, specifically because PCRE and others did not use real automata and he needed a way to do arbitrary regex search safely.

Re: Details of the Cloudflare outage on July 2, 2019

#97

Post by clinta »

foota wrote: post #95

What about: WAF cpu usage wasn't isolated from the ability to serve requests? This would allow requests that don't go throwugh WAF to be able to proceed as usual.

A firewall that fails open sounds like a terrible plan.

As far as problems go, an outage is preferable to a breach.

Re: Details of the Cloudflare outage on July 2, 2019

#98

Post by joncrane »

I used to be really into regex and I'm now rusty, but wouldn't the desired representation of .* .* =.* be something closer to [^=]\* [^=]\* =[^=]\* ?

I feel like it could be optimized further but this would be the first step, and wouldn't most experienced regex authors use that from the beginning, nipping the whole backtracking problem in the bud and making the regex much more performant?

Re: Details of the Cloudflare outage on July 2, 2019

#99

Post by closeparen »

blr246 wrote: post #5

Appreciate the detail here. It's a great writeup. Wondering what folks think about one of the changes: 5. Changing the SOP to do staged rollouts of rules in the same manner used for other software at Cloudflare while retaining the ability to do emergency global deployment for active attacks. One concern I'd have is whether or not I'm exercising the global rollout procedure often enough to be confident it works when i…

One way of dealing with this is regular drills. My employer has a cross-cutting rotation that exercises stuff like this weekly.

Re: Details of the Cloudflare outage on July 2, 2019

#100

Post by asdfasgasdgasdg »

hn_throwaway_99 wrote: post #96

One thing that was interesting to me: The outage was caused by a regex that ended up doing a lot of backtracking, which caused PCRE, the regex engine, to essentially handle a runaway expression. This reminded me of a HN post from a couple months back by the author of Google Code Search, and how it worked: https://swtch.com/~rsc/regexp/regexp4.html . Interestingly, he wrote his own regex engine, RE2, specifically beca…

Yes, this is one of the regexp engines the post discusses switching to as a mitigation.
Post reply on HN