Really appreciate the detail and operational insights in this post-mortem. Great work. Also appreciate the heads up and detailed explanation of the issues with backtracking in regexes. Turns out we use those expensive patterns in a few places.
Details of the Cloudflare outage on July 2, 2019
51–60 of 159 posts
Re: Details of the Cloudflare outage on July 2, 2019
#52> Then we moved on to restoring the WAF functionality. Because of the sensitivity of the situation we performed both negative tests (asking ourselves “was it really that particular change that caused the problem?”) and positive tests (verifying the rollback worked) in a single city using a subset of traffic after removing our paying customers’ traffic from that location. Haha, so the free customers are crash test dum…
Or you can say all customers were affected but some localized free-tier customers got the fix first.
If it's free, you're the product.
Overall I think it's a good deal for both users and Cloudflare. Users get a major CDN for free, and instead of paying for it with ads, surveillance or other shady thing, they pay by being beta testers.
Re: Details of the Cloudflare outage on July 2, 2019
#53> Then we moved on to restoring the WAF functionality. Because of the sensitivity of the situation we performed both negative tests (asking ourselves “was it really that particular change that caused the problem?”) and positive tests (verifying the rollback worked) in a single city using a subset of traffic after removing our paying customers’ traffic from that location. Haha, so the free customers are crash test dum…
I used to work as a network engineer for awhile, now do web development. I worked with a number of cloud providers and you always have to roll out any fix carefully even if you're 100% sure (you're never 100% sure) that you've got the fix.
I honestly just assumed that when customer's chose where they would try things outside their lab, it was lower level customers, less busy part of the network, anywhere the impact isn't as serious. That's where the lowest risk is.
Some customers would discuss their own customer's by name as far as "Should we try this change on Customer Y?" And the discussion would work along those lines.
When I started deploying my own software, I just assumed anything that I was deploying to for free was a sort of "lab light" for them. I also don't mind, it seems fair.
ANY change outside a lab... is its own experiment.
Re: Details of the Cloudflare outage on July 2, 2019
#54Maybe I'm not the intended audience, but I had to look up the acronym WAF since it came up a lot in this article. I'm assuming it's "web application firewall"?
Yes. Web Application Firewall.
Re: Details of the Cloudflare outage on July 2, 2019
#55Sometimes I wish Cloudflare failed more often so I could read more of these postmortems...
I literally LOLed.
Re: Details of the Cloudflare outage on July 2, 2019
#56Here's their What Went Wrong : 1. An engineer wrote a regular expression that could easily backtrack enormously. 2. A protection that would have helped prevent excessive CPU use by a regular expression was removed by mistake during a refactoring of the WAF weeks prior—a refactoring that was part of making the WAF use less CPU. 3. The regular expression engine being used didn’t have complexity guarantees. 4. The test…
I don't see the relevance of how regexes are written to the problem they had. The engineer didn't typo the regex, or have a hard time understanding what it would match. Instead, they didn't understand the runtime performance of the regex, as it was implemented in their particular system. No amount of syntax can change that.
No amount of syntax can change that
A framework that allows well-written, "normal" code to parse out what you want, can produce something easier to understand and maintain, surfacing this type of bug in a more obvious way.
Cryptic syntax is the main reason I avoid regexes (particularly complex ones).
Too much obfuscation between the code you write and the steps your program will take. Granted, my concern doesn't apply to master craftsmen who truly understand the nuances of the tool, but in the real world those are few and far between.
ps. I get there was a lot more going on in this postmortem than just one rogue regex.
Re: Details of the Cloudflare outage on July 2, 2019
#57I’d like to imagine that as long as we live, poorly written regular expressions will continue to be the cause of breaking issues.
StackOverflow also had an outage a few years ago that was caused by exponential blow-up of a backtracking regular expression.
https://stackstatus.net/post/147710624694/outage-postmortem-...
Re: Details of the Cloudflare outage on July 2, 2019
#58> Then we moved on to restoring the WAF functionality. Because of the sensitivity of the situation we performed both negative tests (asking ourselves “was it really that particular change that caused the problem?”) and positive tests (verifying the rollback worked) in a single city using a subset of traffic after removing our paying customers’ traffic from that location. Haha, so the free customers are crash test dum…
Seems fair. You have to roll it out to someone first, so why not roll it out to the users who are not paying for their service.
Re: Details of the Cloudflare outage on July 2, 2019
#59I’d like to imagine that as long as we live, poorly written regular expressions will continue to be the cause of breaking issues.
[deleted]
Re: Details of the Cloudflare outage on July 2, 2019
#60Sometimes I wish Cloudflare failed more often so I could read more of these postmortems...
Please no.
The last time they had a global problem, everyone scrambled for more than a week. (Cloudbleed)
This 30-minute global outage was pretty nasty, but not anywhere near as awful. Timing helped, as nothing truly critical was affected. (There are some extremely high-volume sporting events which, if affected even just for few minutes, can have a direct impact on the bottom line.)
I do not wish to see more of these. Cloudbleed gave me two weeks of headache and an indigestion problem. This one did basically nothing. If there is a happy middle ground between the two, I am not exactly thrilled at finding out what it is.