Might be late, but has anyone in CloudFlare tried to switch away from regex to something more efficient and powerful?
Tools like re2c can convert 100s of regexs and CFG into a single optimized state machine (which includes no back tracking, as far as I remember). It should easily handle 10s of millions transactions per second per core if the complete state machine fits into the CPU level 3 cache (or lower), with a bit of optimization.
Details of the Cloudflare outage on July 2, 2019
61–70 of 159 posts
Re: Details of the Cloudflare outage on July 2, 2019
#62>A protection that would have helped prevent excessive CPU use by a regular expression was removed by mistake during a refactoring of the WAF weeks prior—a refactoring that was part of making the WAF use less CPU.
Faster karma than normal i think.
Re: Details of the Cloudflare outage on July 2, 2019
#63Obligatory JWZ quote:
Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.
Re: Details of the Cloudflare outage on July 2, 2019
#64More general questions I would consider asking:
1. It appears there was a safe path with more safety and scrutiny, and a fast path with less. In this case, over time, the fast path became routine. Are there other places where this pattern could develop or has already developed? Is this tradeoff between speed and scrutiny actually necessary? (ie could you have urgent updates reach production faster but actually receive more scrutiny/more testing, even if that happens after the fact?)
2. In a similar vein, if the system has a failsafe configuration (eg only changes that have passed the full barnyard, or configurations that have been running safely for more than a certain amount of time), would it be plausible to automatically roll servers back to that configuration if they remain unresponsive for a certain amount of time?
3. It seems as though there are multiple points (big WAF refactor, credential expiry, internal services dependent on working prod) where a sufficiently cynical engineer would say "I bet there's something here that could, if not bring down the site, at least ruin someone's day". Is there a suitable voice for this kind of cynicism? Eg, a red team or similar? If you were Murphy's Law incarnate, messing with Cloudflare's systems to achieve maximum mischief, where would you start?
4. I get the sense that there are many reliable and well-tested layers of safety, but is it common to test what happens if they fail anyway? Eg: let's pretend Cloudflare just got knocked out globally by a wizard spell, what do we do? Or let's say our staged rollout system gets completely bypassed because of solar flares, how bad is it? Beyond developing a procedure or training for these kinds of situations, are they actively simulated or practiced?
If anything, I'd guess the root root cause here is a success failure, where the system has been so reliable for so long that the main reactions to it failing are disbelief and unpreparedness. I'm sure it wasn't funny at the time, but it gives me a chuckle to imagine the SREs speculating about Mossad quantum-tunnelling 0days or something because the idea of everything falling over on its own is so unthinkable. Meanwhile, those of us without so many 9s would jump straight to "I probably broke it again."
Re: Details of the Cloudflare outage on July 2, 2019
#65> Then we moved on to restoring the WAF functionality. Because of the sensitivity of the situation we performed both negative tests (asking ourselves “was it really that particular change that caused the problem?”) and positive tests (verifying the rollback worked) in a single city using a subset of traffic after removing our paying customers’ traffic from that location. Haha, so the free customers are crash test dum…
I used to work as a network engineer for awhile, now do web development. I worked with a number of cloud providers and you always have to roll out any fix carefully even if you're 100% sure (you're never 100% sure) that you've got the fix. I honestly just assumed that when customer's chose where they would try things outside their lab, it was lower level customers, less busy part of the network, anywhere the impact i…
Lowest risk, yes but not bulletproof.
Smaller customers don't have the same web traffic, which may not be enough to trip any given failure scenario. One could imagine that the backtracking in an onerous regexep is only triggered with a sufficiently large customer that has a path that is especially difficult to match.
With staged rollout and without a "fast" deploy procedure, by the time it hits the larger customers, it's already been deployed to some percentage of the fleet - and then you still have a problem, with a significant proportion of your fleet.
Staged rollouts are an entirely reasonable risk mitigation idea, mind you, and not one I'm even arguing against.
My point is that unfortunately it's no panacea, especially at scale. Which is what makes this all an experiment.
Re: Details of the Cloudflare outage on July 2, 2019
#66I'm a relatively novice regex user. Could anyone explain to me why someone might use an expression like `.(?:.=.*)` ? What is the meaning of the group if it's boundary could be placed in any number of places? Hope that makes sense.
Re: Details of the Cloudflare outage on July 2, 2019
#67I’d like to imagine that as long as we live, poorly written regular expressions will continue to be the cause of breaking issues.
StackOverflow also had an outage a few years ago that was caused by exponential blow-up of a backtracking regular expression. https://stackstatus.net/post/147710624694/outage-postmortem-...
That blow up is quadratic, not exponential: "This is not classic catastrophic backtracking (talk on backtracking) (performance is O(n²), not exponential, in length), but it was enough."
Re: Details of the Cloudflare outage on July 2, 2019
#68I'm a relatively novice regex user. Could anyone explain to me why someone might use an expression like `. (?:. =.*)` ? What is the meaning of the group if it's boundary could be placed in any number of places? Hope that makes sense.
It can be useful depending on how the engine handles the match. The non-capturing group is the important part, the .* is just there so that the 'full match' isn't just an empty string, but contains the whole line the rule is being run against
Re: Details of the Cloudflare outage on July 2, 2019
#699. We had difficulty accessing our own systems because of the outage and the bypass procedure wasn’t well trained on. Suggestion for future, learned from bitter experience: separate your control plane from your data plane. In this case, make sure that the tools you use to manage your infrastructure don't depend on that infrastructure being functional. That way you won't have to remember how to use a bypass procedure…
Yeah, that is true in most cases. However, here is Cloudflare was using Cloudflare on dash.cloudflare.com as well.
This calls for not using Cloudflare for their web dashboard.
Re: Details of the Cloudflare outage on July 2, 2019
#70So in response to a catastrophic failure due to testing in prod, they're going to push out a brand new regex engine with an ETA of 2 weeks. Can anyone say testing in prod? The constant use of 'I' and 'me' (19 occurrences in total) deeply tarnishes this report, and repeatedly singling out a responsible engineer, nameless or not, is a failure in its own right. This was a collective failure, any individual identity is t…
Personal attacks aren't allowed on HN, and please don't post in the flamewar style here generally.