Details of the Cloudflare outage on July 2, 2019

#1

Post by jgrahamc » Fri, Jul 12, 2019, 3:46 PM UTC

Details of the Cloudflare outage on July 2, 2019
blog.cloudflare.com

Re: Details of the Cloudflare outage on July 2, 2019

#2

Post by woliveirajr » Fri, Jul 12, 2019, 3:50 PM UTC

> Unfortunately, last Tuesday’s update contained a regular expression that backtracked enormously and exhausted CPU used for HTTP/HTTPS serving.

One of those cases where they had 1 problem, used regular expression and ended up with 2 problems ?

Edit: I really like how much information is given by CloudFlare. 11 points in the "what went wrong analysis" is how every root-cause analysis should be done.

Re: Details of the Cloudflare outage on July 2, 2019

#3

Post by helper » Fri, Jul 12, 2019, 4:22 PM UTC

This is a fantastic postmortem. Thanks jgc!

Can you share any more details about the protection to prevent excessive cpu usage by a regular expression that was accidentally removed?

Re: Details of the Cloudflare outage on July 2, 2019

#4

Post by toomuchtodo » Fri, Jul 12, 2019, 5:05 PM UTC

woliveirajr wrote: post #2

> Unfortunately, last Tuesday’s update contained a regular expression that backtracked enormously and exhausted CPU used for HTTP/HTTPS serving. One of those cases where they had 1 problem, used regular expression and ended up with 2 problems ? Edit: I really like how much information is given by CloudFlare. 11 points in the "what went wrong analysis" is how every root-cause analysis should be done.

Somewhat humorous, as someone [1] (congrats /u/fossuser!) mentioned this failure scenario in the thread about Twitter being down yesterday.

"Pushing bad regex to production, chaos monkey code causing cascading network failure, etc.", in response to a comment from someone who previously worked at Cloudflare.

[1] https://news.ycombinator.com/item?id=20415608

Re: Details of the Cloudflare outage on July 2, 2019

#5

Post by blr246 » Fri, Jul 12, 2019, 5:10 PM UTC

Appreciate the detail here. It's a great writeup. Wondering what folks think about one of the changes:

  5. Changing the SOP to do staged rollouts of rules in
     the same manner used for other software at Cloudflare
     while retaining the ability to do emergency global
     deployment for active attacks.

One concern I'd have is whether or not I'm exercising the global rollout procedure often enough to be confident it works when it's needed. Of the hundreds of WAF rule changes rolled out every month, how many are global emergencies?

It's a fact of managing process that branches are liability and the hot path is the thing that will have the highest level of reliability. I wonder if anyone there has concerns about diluting the rapid response path (the one having the highest associated risk) by making this process change.

edit: fix verbatim formatting

Re: Details of the Cloudflare outage on July 2, 2019

#6

Post by cordite » Fri, Jul 12, 2019, 5:12 PM UTC

That was a fantastic demonstration of what backtracking meant. Thank you John for your in depth description of what went wrong.

As a follow up, would something like `[^=]=.` be a better capture group regex?

Re: Details of the Cloudflare outage on July 2, 2019

#7

Post by riobard » Fri, Jul 12, 2019, 5:17 PM UTC

Note: Golang's stdlib regexp (https://golang.org/pkg/regexp/) is guaranteed to run in time linear to the size of input. Russ Cox has a detailed article https://swtch.com/~rsc/regexp/regexp1.html

Re: Details of the Cloudflare outage on July 2, 2019

#8

Post by jefftk » Fri, Jul 12, 2019, 5:33 PM UTC

riobard wrote: post #7

Note: Golang's stdlib regexp ( https://golang.org/pkg/regexp/ ) is guaranteed to run in time linear to the size of input. Russ Cox has a detailed article https://swtch.com/~rsc/regexp/regexp1.html

Golang's regexp is derived from RE2, referenced in footnote four of the post.

Re: Details of the Cloudflare outage on July 2, 2019

#9

Post by ch » Fri, Jul 12, 2019, 5:39 PM UTC

blr246 wrote: post #5

Appreciate the detail here. It's a great writeup. Wondering what folks think about one of the changes: 5. Changing the SOP to do staged rollouts of rules in the same manner used for other software at Cloudflare while retaining the ability to do emergency global deployment for active attacks. One concern I'd have is whether or not I'm exercising the global rollout procedure often enough to be confident it works when i…

They still retain the global rollout for the other use cases detailed in the write up, so its generally tested, though not for this one use case as you point out. I suspect the tradeoff is reasonable, however having a short pre-stage deploy before global in all cases would be a more conservative option that would prevent an emergent push from becoming an even bigger emergency!

Re: Details of the Cloudflare outage on July 2, 2019

#10

Post by jakejarvis » Fri, Jul 12, 2019, 5:43 PM UTC

Always appreciate the transparency from you and Cloudflare. :)

My main fright during this outage wasn't really the outage itself, but the fact that I couldn't log into the dashboard and simply click the orange cloud to bypass Cloudflare in the meantime. I'm assuming that this is now covered by this mitigation:

>> 6. Putting in place an emergency ability to take the Cloudflare Dashboard and API off Cloudflare's edge.

If so, and if this would have prevented the dashboard outage even during the WAF fiasco, this is a huge comfort to me. Just curious, though: how far can you really go in separating Cloudflare "the interface" from Cloudflare "the network?"

And in general, what does everyone on HN think about mission-critical companies using their own infrastructure and being their own customer? Especially when the alternative is using a competitor?