Live data from Hacker News

Details of the Cloudflare outage on July 2, 2019

blog.cloudflare.com

141–150 of 159 posts

Re: Details of the Cloudflare outage on July 2, 2019

#141

> Unfortunately, last Tuesday’s update contained a regular expression that backtracked enormously and exhausted CPU used for HTTP/HTTPS serving. One of those cases where they had 1 problem, used regular expression and ended up with 2 problems ? Edit: I really like how much information is given by CloudFlare. 11 points in the "what went wrong analysis" is how every root-cause analysis should be done.

I agree this is an awesome post and a really great example of how every Root Cause Analysis needs to be done. I am also impressed by their incident response.

Re: Details of the Cloudflare outage on July 2, 2019

#142
post #11
post #5

Appreciate the detail here. It's a great writeup. Wondering what folks think about one of the changes: 5. Changing the SOP to do staged rollouts of rules in the same manner used for other software at Cloudflare while retaining the ability to do emergency global deployment for active attacks. One concern I'd have is whether or not I'm exercising the global rollout procedure often enough to be confident it works when i…

Yep, that's the exact bullet point I was writing a response on. Security and abuse are of course special little snowflakes, with configs that need to be pushed very fast, contrary to all best practices for safe deployments of globally distributed systems. An anti-abuse rule that takes three days to roll out might as well not exist. The only way this makes sense is if they mean that there'll be a staged rollout of som…

> Security and abuse are of course special little snowflakes, with configs that need to be pushed very fast, contrary to all best practices for safe deployments of globally distributed systems.

Once upon a time, I worked on a system where many values which would otherwise be statically defined in similar systems where instead put into a database table. This particular system didn't have a proper testing and deployment pipeline set up, so whereas a normal system would just change the static value at some hard-coded point in the code and quickly roll it out, this system needed to keep it in the database so that it would be changeable in between manual deployments (months or even years apart). The ability to change the value facing the user by changing the value in the database inflated the time it took to test a release, thus exacerbating the amount of time it took to release a new version, but well, it worked.

My point is that if security and abuse rules need to be rolled out quickly, then the system needs security and abuse systems where the entire range of security and abuse configurations (i.e. their types) are a testable part of the original pipeline. Then the configurations can safely be changed on the fly, so long as the changes type-check.

It's easy to understand why it's never been built though - you'd need both a security background and a Haskell-ish/type-theory kind of background. Best of luck finding people like that.

Re: Details of the Cloudflare outage on July 2, 2019

#143
post #67
post #57

Earlier quoted context omitted.

StackOverflow also had an outage a few years ago that was caused by exponential blow-up of a backtracking regular expression. https://stackstatus.net/post/147710624694/outage-postmortem-...

That blow up is quadratic, not exponential: "This is not classic catastrophic backtracking (talk on backtracking) (performance is O(n²), not exponential, in length), but it was enough."

Some years ago I tried to create an exponential time regex in Perl, and only managed a quadratic time one after a bit of experimenting.

But, as you said, quadratic is often already fatal on realistic data.

Re: Details of the Cloudflare outage on July 2, 2019

#144
Amazing postmortem! I have a question: I see a lot of software / process solutions to avoid this to repeat in the future. What about the Human factor?

- what happened to the engineer(s) responsible for that event? They must feel really bad RN, how do you handle this situation?

- on a more general point, how do you train individuals to ensure this particular event does not reproduce?

Edit: formatting.

Re: Details of the Cloudflare outage on July 2, 2019

#145
post #106

One thing that was interesting to me: The outage was caused by a regex that ended up doing a lot of backtracking, which caused PCRE, the regex engine, to essentially handle a runaway expression. This reminded me of a HN post from a couple months back by the author of Google Code Search, and how it worked: https://swtch.com/~rsc/regexp/regexp4.html . Interestingly, he wrote his own regex engine, RE2, specifically beca…

The problem is that a deterministic regex engine (deterministic finite automata or DFA) is strictly less powerful than a non-deterministic one (NFA). DFA's can't backtrack, for example. In addition, DFA's can be quite a bit slower for certain inputs and matches.

The problem is not DFA vs NFA.

“regular expression” has different meaning in programming context and formal language context. Regular expressions in regex libraries do more than match regular languages.

PCRE can recognize also all context free languages and some subset of context-sensitive languages. Just having backreferences makes the problem NP-hard.

Re: Details of the Cloudflare outage on July 2, 2019

#147
post #28
post #17

Earlier quoted context omitted.

Your response highlights a good idea to mitigate the risk I was trying to highlight in mine. They want to have a rapid response path (little to no delay using staging envs) to respond to emergencies. The old SOP allowed all releases to use the emergency path. By not using it in the SOP anymore, I'd be concerned that it would break silently from some other refactor or change. Your notion is to maintain the emergency r…

Right. The emergency path is either something you end up using always, or something you use so rarely that it gets eaten by bit-rot before it gets ever used[0]. So I think we're in full agreement on your original point. This was just an attempt to parse a working policy out of that bullet point. [0] My favorite example of this had somebody accidentally trigger an ancient emergency config push procedure. It worked, ma…

My personal rule is that any code which doesn't get exercised at least weekly is untrustworthy. I once inherited a codebase with a heavy, custom blue-green deploy system (it made sense for the original authors). While we deployed about once a week, we set up CI to test the deployment every day.

Cold code is dead code.

Re: Details of the Cloudflare outage on July 2, 2019

#148
post #5

Appreciate the detail here. It's a great writeup. Wondering what folks think about one of the changes: 5. Changing the SOP to do staged rollouts of rules in the same manner used for other software at Cloudflare while retaining the ability to do emergency global deployment for active attacks. One concern I'd have is whether or not I'm exercising the global rollout procedure often enough to be confident it works when i…

The main problem is that their Regex library doesn't have a recrusion limit. I'm honestly amazed they've been able to scale Lua scripts to the point they can use it as a global WAF. Knowing this, it may be easy to create attacks against their filters. My takeaway is that it's time to move to a custom solution using a more flexible language. A simple async watchdog on total rule execution time would have prevented thi…

I'm interested in why they wouldn't use LPeg instead. Those seem a lot easier to compose, reason about and debug; plus they have restricted backtracking.

Re: Details of the Cloudflare outage on July 2, 2019

#149

Earlier quoted context omitted.

The main problem is that their Regex library doesn't have a recrusion limit. I'm honestly amazed they've been able to scale Lua scripts to the point they can use it as a global WAF. Knowing this, it may be easy to create attacks against their filters. My takeaway is that it's time to move to a custom solution using a more flexible language. A simple async watchdog on total rule execution time would have prevented thi…

I am wondering why you are being downvoted. This outage could have been prevented with better deployment procedures too. For example my company (nowhere near the scale of Cloudflare) does progressive deployments. New code is deployed only to a handful machines first, and then as the hours pass and checks remain green it propagates to the rest of the server fleet. Full deployment takes 24 hours. We never had code brea…

Confused what this response is trying to say? Did you read the whole post? They addressed exactly those two things and explained how they're fixing them. You're just repeating part the blog post essentially; which is why I wonder if you finished reading it.

Re: Details of the Cloudflare outage on July 2, 2019

#150
Great post! Can’t think of many companies that would spend this much time explaining regex.

I was affected by this outage, but I really appreciate Cloudflare taking the time to explain the problem in this much detail. Given their own systems were affected, I’m surprised they mitigated as fast as they did.

Post reply on HN