Seems like a terrible idea to deploy changes to such a vital piece of their software GLOBALLY without some sort of rollout procedure.
Cloudflare outage caused by bad software deploy
61–70 of 137 posts
Re: Cloudflare outage caused by bad software deploy
#62Probably for the kind of work they are doing avoid regex? Or at least the very complicated modern regex (simple autonoma that you can compile in advance might be ok)
Re: Cloudflare outage caused by bad software deploy
#63What sort of regular expression pitfalls can cause this sort of CPU utilization? I know they're possible but I am curious about specific examples of something similar to what caused Cloudflare's issue here.
For examples check the bugs that were duped into https://bugs.chromium.org/p/v8/issues/detail?id=430
Re: Cloudflare outage caused by bad software deploy
#64What sort of regular expression pitfalls can cause this sort of CPU utilization? I know they're possible but I am curious about specific examples of something similar to what caused Cloudflare's issue here.
DoS with regex is a thing: https://www.owasp.org/index.php/Regular_expression_Denial_of... StackOverflow had a similar case a while back: https://stackstatus.net/post/147710624694/outage-postmortem-...
HN discussion of the StackOverflow incident: https://news.ycombinator.com/item?id=12131909
Re: Cloudflare outage caused by bad software deploy
#65What sort of regular expression pitfalls can cause this sort of CPU utilization? I know they're possible but I am curious about specific examples of something similar to what caused Cloudflare's issue here.
Some regex languages allow backtracking, and backtracking is usually the thing that causes regexes to blow up in resource cost: https://www.regular-expressions.info/catastrophic.html
You probably want a regex engine that runs in linear time:
* Google's RE2 https://github.com/google/re2/wiki/WhyRE2
* https://github.com/laurikari/tre/
There is a good series of articles about the problem: https://swtch.com/~rsc/regexp/regexp3.html
I would strongly recommend deploying such a regular expression matcher to avoid problems like this. There are examples in the above article that you can use to test anything in your production deployment that accepts regular expressions to see how well it copes.
Re: Cloudflare outage caused by bad software deploy
#66For the size and importance of Cloudflare some insights to a couple questions would be nice: 1. Why are WAF rules not progressively deployed since there's already a system to do so? 2. Maybe there should also be a testing environment that receives a mirror of production traffic before deployments reach real users? (I understand the WAF change was not set to take action, but a separate environment would be less likely…
Cloudflare does have test colos that use a subset of real network traffic for testing. It's actually the primary testing methodology, and the employees are usually some of the first people forced through test updates.
This release wasn't meant to go out, and the fact it did means it would have bypassed the test environments either way.
Re: Cloudflare outage caused by bad software deploy
#67Peculiar that eastdakota (Cloudflare's CEO) doesn't seem to be tweeting at the Cloudflare team responsible for this, telling them they should be ashamed and are guilty of malpractice.
When it was Verizon that took down the internet he felt it was appropriate to do that to the Verizon teams, after all.
edit: right after posting this comment, he did tweet the following: https://twitter.com/eastdakota/status/1146196836035620864
> I'd say both we and Verizon deserve to be ashamed.
As well as this: https://twitter.com/eastdakota/status/1146170209780113408
> Our team should be and is ashamed. And we deserve criticism. ...
I still don't think that publicly shaming anyone is a good leadership style nor is it a good way to motivate people to perform better in the future, but kudos for the self-awareness, at least.
Re: Cloudflare outage caused by bad software deploy
#68Earlier quoted context omitted.
Some regex languages allow backtracking, and backtracking is usually the thing that causes regexes to blow up in resource cost: https://www.regular-expressions.info/catastrophic.html
You probably want a regex engine that runs in linear time: * Google's RE2 https://github.com/google/re2/wiki/WhyRE2 * https://github.com/laurikari/tre/ There is a good series of articles about the problem: https://swtch.com/~rsc/regexp/regexp3.html I would strongly recommend deploying such a regular expression matcher to avoid problems like this. There are examples in the above article that you can use to test anythi…
Might have been a misdirect reply (although useful), but yeah, agree, linear-time regex engines are generally a much better idea.
Re: Cloudflare outage caused by bad software deploy
#69If a single regex can take down the Internet for a half hour, that's definitely not good -- for a class of errors that can be easily prevented, tested, etc. The timing is unfortunate too, after calling out Verizon for lack of due process and negligence. I'm sure they have an undo or rollback for deployments but probably worth investing into further. They also need to resolve the catch-22 where people could not login…
> The timing is unfortunate too, after calling out Verizon for lack of due process and negligence.
Nonetheless, Verizon could take a leaf out of their responsiveness and transparency book.
Re: Cloudflare outage caused by bad software deploy
#70Is there a usage error in the first sentence or has English lost the blog / blog post distinction?