I’d like to imagine that as long as we live, poorly written regular expressions will continue to be the cause of breaking issues.
Details of the Cloudflare outage on July 2, 2019
31–40 of 159 posts
Re: Details of the Cloudflare outage on July 2, 2019
#32> In the last few years we have seen a dramatic increase in vulnerabilities in common applications. This has happened due to the increased availability of software testing tools, like fuzzing for example (we just posted a new blog on fuzzing here).
So security/debugging tools increased the number of [discovered/exploited] vulnerabilities, because developers don't use them. Only malware developers and third-party security researchers take the time to test security.
Re: Details of the Cloudflare outage on July 2, 2019
#33> In the last few years we have seen a dramatic increase in vulnerabilities in common applications. This has happened due to the increased availability of software testing tools, like fuzzing for example (we just posted a new blog on fuzzing here). So security/debugging tools increased the number of [discovered/exploited] vulnerabilities, because developers don't use them. Only malware developers and third-party secu…
Yup, unless you're a seriously security or stability focused company you don't even use basic stuff like static analysis, let alone fuzzing. Tools for these are often either expensive, hard to use or both.
Re: Details of the Cloudflare outage on July 2, 2019
#34> Then we moved on to restoring the WAF functionality. Because of the sensitivity of the situation we performed both negative tests (asking ourselves “was it really that particular change that caused the problem?”) and positive tests (verifying the rollback worked) in a single city using a subset of traffic after removing our paying customers’ traffic from that location. Haha, so the free customers are crash test dum…
[deleted]
Re: Details of the Cloudflare outage on July 2, 2019
#35Here's their What Went Wrong : 1. An engineer wrote a regular expression that could easily backtrack enormously. 2. A protection that would have helped prevent excessive CPU use by a regular expression was removed by mistake during a refactoring of the WAF weeks prior—a refactoring that was part of making the WAF use less CPU. 3. The regular expression engine being used didn’t have complexity guarantees. 4. The test…
I don't see the relevance of how regexes are written to the problem they had. The engineer didn't typo the regex, or have a hard time understanding what it would match.
Instead, they didn't understand the runtime performance of the regex, as it was implemented in their particular system. No amount of syntax can change that.
Re: Details of the Cloudflare outage on July 2, 2019
#36> Then we moved on to restoring the WAF functionality. Because of the sensitivity of the situation we performed both negative tests (asking ourselves “was it really that particular change that caused the problem?”) and positive tests (verifying the rollback worked) in a single city using a subset of traffic after removing our paying customers’ traffic from that location. Haha, so the free customers are crash test dum…
Or you can say all customers were affected but some localized free-tier customers got the fix first.
Re: Details of the Cloudflare outage on July 2, 2019
#37Awesome write up as usual John. I'm no expert so I was wondering: "In the initial moments of the outage there was speculation it was an attack of some type we’d never seen before." - Is there a reason you would go to this first vs checking the last lot of code deploys first/and or at the same time?
> At 13:37 TeamCity built the rules and ran the tests, giving it the green light.
I assume this is why :D
Re: Details of the Cloudflare outage on July 2, 2019
#38This is such a fantastic blogpost. I love the part about regex backtracking. Great job CF.
Re: Details of the Cloudflare outage on July 2, 2019
#39> Then we moved on to restoring the WAF functionality. Because of the sensitivity of the situation we performed both negative tests (asking ourselves “was it really that particular change that caused the problem?”) and positive tests (verifying the rollback worked) in a single city using a subset of traffic after removing our paying customers’ traffic from that location. Haha, so the free customers are crash test dum…
Or you can say all customers were affected but some localized free-tier customers got the fix first.
In this case yes, however they also indicate this is how they do their staged rollouts in general. So if they are releasing any other software update that goes through the staged rollout free customers are tested first. If that change broke something, free customers get that first. Which seems fair to me.
Re: Details of the Cloudflare outage on July 2, 2019
#40> The Lua WAF uses PCRE internally and it uses backtracking for matching and has no mechanism to protect against a runaway expression. More on that and what we're doing about it below.
We run a WAF based on LuaJIT in resty. Just to be clear, the resty interface to PCRE does provide a DFA mode. Furthermore, Zhang actually ported RE2 (see other comments here) to C as sregex, which is usable from Lua as a c module regardless if it runs in resty or a custom Lua app.
> Switching to either the re2 or Rust regex engine which both have run-time guarantees. (ETA: July 31)
Not addressed at Cloudflare, since they had a defense in place. But just in case anyone else is running a similar thing in Lua.
And:
> In the longer term we are moving away from the Lua WAF that I wrote years ago.
Then sregex might be the perfect fit here. Though Rust is technically safer. Depends on what longer term means.