Details of the Cloudflare outage on July 2, 2019 • Hacker News Board

Re: Details of the Cloudflare outage on July 2, 2019

#81

Post by f2f » Fri, Jul 12, 2019, 10:40 PM UTC

Twirrim wrote: post #71

One thing set of alarm bells in my head from an operational perspective: > Switching to either the re2 or Rust regex engine which both have run-time guarantees. (ETA: July 31) That's short timescales for quite a significant change. I know it's just replacing a piece of automation with one that does the same task, but the guts are all changing and all automation introduces some level of instability, and a bunch of unk…

RE2 is bullet proof. It is the defacto re engine used by anyone looking to deploy regexes to the wild (used in the now defunct google code search, for example). It has a track record. Russ Cox, its author, was affiliated with Ken Thompson from very early in his career.

Rust also has a good pedigree for not being faulty. BurntSushi, the author of rust's regex crate also has a good pedigree...

We switched to RE2 for a massive project 2 years ago and haven't looked back. It is a massive improvement in peace of mind.

If anything, I'm surprised that JGC has allowed the use of PCRE in production and on live inputs...

Re: Details of the Cloudflare outage on July 2, 2019

#82

Post by jalada » Fri, Jul 12, 2019, 10:49 PM UTC

cordite wrote: post #6

That was a fantastic demonstration of what backtracking meant. Thank you John for your in depth description of what went wrong. As a follow up, would something like `[^=] =. ` be a better capture group regex?

Yes. I think HN stole your asterisks. You meant:

  /.*=.*/ becomes /[^=]*=.*/

That is, zero or more 'not-equals-sign-characters', followed by an equals sign.

Where the first regex is 57 steps for x=xxxxxxxxxxxxxxxxxxxxxxxx, the second is just 7.

Avoid using greedy .* for backtracking regex engines! Give your greedy regex engine the hints it needs to do what it does best.

Re: Details of the Cloudflare outage on July 2, 2019

#83

Post by JakeTheAndroid » Fri, Jul 12, 2019, 11:21 PM UTC

londons_explore wrote: post #76

Cloudflare lets their customers write their own WAF regex rules right? And those rules still get run on every box on cloudflares edge network with HTTP requests from strangers on the internet right? So how come this didn't get triggered by a customer first? Perhaps it did get triggered by a customer first, but that customer didn't get too much traffic of the URL which triggers the issue, and that box got one thread s…

They allow a limited subset of rules, with strict parameters of what logic is allowed. Unless you do something fancy with workers.

Also, the protection for this was removed in a recent update before the incident, so it wouldn't have had an impact if a customer did this until that protect was removed. So maybe a few weeks earlier they might have started seeing some problems. But again, I am pretty sure the logic in the rule that caused the issue isn't available to customers.

Re: Details of the Cloudflare outage on July 2, 2019

#84

Post by peterwwillis » Fri, Jul 12, 2019, 11:24 PM UTC

hyperpape wrote: post #35

peterwwillis wrote: post #25

Here's their What Went Wrong : 1. An engineer wrote a regular expression that could easily backtrack enormously. 2. A protection that would have helped prevent excessive CPU use by a regular expression was removed by mistake during a refactoring of the WAF weeks prior—a refactoring that was part of making the WAF use less CPU. 3. The regular expression engine being used didn’t have complexity guarantees. 4. The test…

I don't see the relevance of how regexes are written to the problem they had. The engineer didn't typo the regex, or have a hard time understanding what it would match. Instead, they didn't understand the runtime performance of the regex, as it was implemented in their particular system. No amount of syntax can change that.

By writing regexs by hand, you can accidentally introduce an obviously backtracking pattern such as * .=. *. By programmatically composing them, a program can analyze each regex group to find simple problems, and then combine them in ways that will avoid backtracking.

This isn't even why you should compose them programmatically, though. Perl allows you to compose a regex with in-line comments (https://perldoc.perl.org/perlfaq6.html#How-can-I-hope-to-use...), but it's still a hand-crafted regex, which is error-prone, much like composing code by hand. If you can get a machine to generate it for you, you avoid unintentional human-introduced bugs, as well as make it easier to read and reason about.

If you have a ton of regex's, or they are super important to your business, you should consider not editing them by hand. There's only so much test cases can do to prevent bugs.

Re: Details of the Cloudflare outage on July 2, 2019

#85

Post by stevens32 » Fri, Jul 12, 2019, 11:26 PM UTC

For the regex novices here, would anyone mind explaining what that pattern is meant to match? More specifically, what `.(?:.=.*)` is meant to do?

Re: Details of the Cloudflare outage on July 2, 2019

#86

Post by glangdale » Fri, Jul 12, 2019, 11:29 PM UTC

Well, if I still worked on Hyperscan, this would be my "what am I, a potted plant?" moment. I think Cloudflare is pretty determined to avoid x86-only implementations of anything, though.

It's entertaining to see people making the same mistakes that have been widely known about in network security well before there was Hyperscan, RE2, etc.

Re: Details of the Cloudflare outage on July 2, 2019

#87

Post by Twirrim » Fri, Jul 12, 2019, 11:29 PM UTC

f2f wrote: post #81

Twirrim wrote: post #71

One thing set of alarm bells in my head from an operational perspective: > Switching to either the re2 or Rust regex engine which both have run-time guarantees. (ETA: July 31) That's short timescales for quite a significant change. I know it's just replacing a piece of automation with one that does the same task, but the guts are all changing and all automation introduces some level of instability, and a bunch of unk…

RE2 is bullet proof. It is the defacto re engine used by anyone looking to deploy regexes to the wild (used in the now defunct google code search, for example). It has a track record. Russ Cox, its author, was affiliated with Ken Thompson from very early in his career. Rust also has a good pedigree for not being faulty. BurntSushi, the author of rust's regex crate also has a good pedigree... We switched to RE2 for a…

I'm absolutely not denying that RE2 is great. Not in the slightest. I even agree with their idea to switch towards it or the Rust one.

Changing anything brings an element of risk, and changing quickly to it, even more so, which is essentially what they're proposing doing. That's where my concern lies.

Their current approach clearly has issues, but it has been running in production for several years now and those issues are fully understood, engineers know how to debug them, and there's a lot of institutional knowledge around covering them. They've put a series of protective measures in place following the incident that takes out one of the more significant risks. That gives them breathing space to evaluate and verify their options, carry out smaller scale experiments, train up engineers across the company around any relevant changes etc. There is no reason to go _fast_

Re: Details of the Cloudflare outage on July 2, 2019

#88

Post by charrondev » Fri, Jul 12, 2019, 11:30 PM UTC

regnerba wrote: post #39

pvg wrote: post #36

Earlier quoted context omitted.

Or you can say all customers were affected but some localized free-tier customers got the fix first.

In this case yes, however they also indicate this is how they do their staged rollouts in general. So if they are releasing any other software update that goes through the staged rollout free customers are tested first. If that change broke something, free customers get that first. Which seems fair to me.

In my experience it’s generally best to roll out changes on testing, staging, and then clients in order of how much they pay, especially if you have SLAs with the highest paying customers.

Impact is generally lower, both to the client, and to your bank account.

Re: Details of the Cloudflare outage on July 2, 2019

#89

Post by migueldemoura » Fri, Jul 12, 2019, 11:32 PM UTC

londons_explore wrote: post #76

Cloudflare lets their customers write their own WAF regex rules right? And those rules still get run on every box on cloudflares edge network with HTTP requests from strangers on the internet right? So how come this didn't get triggered by a customer first? Perhaps it did get triggered by a customer first, but that customer didn't get too much traffic of the URL which triggers the issue, and that box got one thread s…

> Cloudflare lets their customers write their own WAF regex rules right?

No, but customers can request a custom WAF rule to be written by Cloudflare engineers specifically for their domain.

Re: Details of the Cloudflare outage on July 2, 2019

#90

Post by [deleted] » Fri, Jul 12, 2019, 11:43 PM UTC

minxomat wrote: post #40

> The Lua WAF uses PCRE internally and it uses backtracking for matching and has no mechanism to protect against a runaway expression. More on that and what we're doing about it below. We run a WAF based on LuaJIT in resty. Just to be clear, the resty interface to PCRE does provide a DFA mode. Furthermore, Zhang actually ported RE2 (see other comments here) to C as sregex, which is usable from Lua as a c module regar…

[deleted]