Details of the Cloudflare outage on July 2, 2019 • Hacker News Board

Re: Details of the Cloudflare outage on July 2, 2019

#151

Post by karmakaze » Sat, Jul 13, 2019, 2:58 PM UTC

TL;DR

Root cause was a bad regex generating excessive backtracking using all CPU on nodes.

The meta-cause is the process workflow:

> But, by design, the WAF doesn’t use this process because of the need to respond rapidly to threats.

The above is in reference to how WAF deployment doesn't use the graduated DOG(fooding)/(guinea)PIG/canary flow.

> We responded quickly to correct the situation and are correcting the process deficiencies that allowed the outage to occur [...]

Live and learn. Not all WAF deployments are emergency rollouts.

Re: Details of the Cloudflare outage on July 2, 2019

#152

Post by posixplz » Sat, Jul 13, 2019, 4:36 PM UTC

dsr_ wrote: post #49

9. We had difficulty accessing our own systems because of the outage and the bypass procedure wasn’t well trained on. Suggestion for future, learned from bitter experience: separate your control plane from your data plane. In this case, make sure that the tools you use to manage your infrastructure don't depend on that infrastructure being functional. That way you won't have to remember how to use a bypass procedure…

Well, as with all things based in technical nuance, it depends on your definitions. Sure, control planes and data planes should be logically separated. But as you build and ship compelling products, your developers will gravitate to using well-built products’ (data plane) resources to build new products.

Imagine an IaaS cloud. It starts will Compute, Networking, Storage (block) and maybe Object Storage/S3. Next comes a fully-managed database product. The Database team may want to leverage the Object Storage data plane in the Database control plane. A year or two down the road, a team building a SaaS application will probably look to use the fully-managed database as it’s one less piece of infrastructure to manage.

To avoid or eliminate these types of delays in resolution, it’s imperative that the product team have a strong understanding of failure modes and dependencies. There’s a lot to be said for building completely isolated foundational services — it’s also a very expensive undertaking. Lastly, it’s possible to build out-of-band/break glass access without compromising security.

(I work at a global cloud but have no familiarity with CloudFlare’s internals.)

Re: Details of the Cloudflare outage on July 2, 2019

#153

Post by hinkley » Sat, Jul 13, 2019, 7:38 PM UTC

Thorrez wrote: post #125

hinkley wrote: post #80

Earlier quoted context omitted.

Taking the safeties off to go faster... yes, you will go faster, but it might be right off a cliff. This is a good lesson on Chesterton’s Fence. I’ve been thinking for a while that we really need the (default behavior) ability to annotate commits after the fact, so that we have a durable commentary that can evolve over time. We should be able to go back and add strongly worded things like “yes this looks broken but i…

Can't you just add comments to the actual code saying those things? I've seen code comments saying those exact things.

There’s an art to that, I’ve seen new code get between the comment and the code, and since the comment is in a separate commit, it’s difficult to go back ten refactorings later to answer why. The most interesting bug fixes I do end up exploiting the commit history. Yes it’s hidden in plain sight, but it’s also more reliable.

These days we treat code as a living breathing thing. No reason we can’t do the same to commits.

Re: Details of the Cloudflare outage on July 2, 2019

#154

Post by jgrahamc » Sun, Jul 14, 2019, 4:19 PM UTC

pvg wrote: post #47

jgrahamc wrote: post #21

Earlier quoted context omitted.

This report is written by me, the CTO of Cloudflare. I say "I" throughout because organizational failings are my responsibilty. If I'd said "we" I imagine you'd be criticizing me for NOT taking responsibility. If you read the report you'd see I do not blame the engineer responsible at all. Not once. I made that perfectly clear.

I wonder if you are able to talk a bit about the development of the Lua-based WAF. I imagine the possible unbounded performance of feeding requests into PCRE must have occurred to you or others at the time - or at least, long before this outage. I don't mean this as some sort of lame 'lol shoulda known better' dunk - stories about technical organizations' decision-making and tradeoff-handling are just more interestin…

I did a talk about this years ago: https://www.youtube.com/watch?v=nlt4XKhucS4

Re: Details of the Cloudflare outage on July 2, 2019

#155

Post by pvg » Sun, Jul 14, 2019, 4:54 PM UTC

jgrahamc wrote: post #154

pvg wrote: post #47

Earlier quoted context omitted.

I wonder if you are able to talk a bit about the development of the Lua-based WAF. I imagine the possible unbounded performance of feeding requests into PCRE must have occurred to you or others at the time - or at least, long before this outage. I don't mean this as some sort of lame 'lol shoulda known better' dunk - stories about technical organizations' decision-making and tradeoff-handling are just more interestin…

I did a talk about this years ago: https://www.youtube.com/watch?v=nlt4XKhucS4

It sounds like one of the primary factors was compatibility with existing (or customer-provided) mod_security rules, if I've understood 1.75x speed hyper-you right.

Re: Details of the Cloudflare outage on July 2, 2019

#156

Post by thurston » Wed, Jul 17, 2019, 4:10 PM UTC

edwintorok wrote: post #77

lkoolma wrote: post #61

Might be late, but has anyone in CloudFlare tried to switch away from regex to something more efficient and powerful? Tools like re2c can convert 100s of regexs and CFG into a single optimized state machine (which includes no back tracking, as far as I remember). It should easily handle 10s of millions transactions per second per core if the complete state machine fits into the CPU level 3 cache (or lower), with a bi…

There is also Ragel [0], but I think that in this context deploying regexes as strings is safer than generating code and deploying that code (unless Ragel could generate webassembly). [0]: http://www.colm.net/open-source/ragel/

Ragel has the advantage that CPU blowups happen at compile time, rather than run-time. Other risks aside, they would have avoided this problem had they been using ragel or something similar to pre-compile their patterns into deterministic machines.

Re: Details of the Cloudflare outage on July 2, 2019

#157

Post by thurston » Wed, Jul 17, 2019, 4:12 PM UTC

edwintorok wrote: post #77

lkoolma wrote: post #61

Might be late, but has anyone in CloudFlare tried to switch away from regex to something more efficient and powerful? Tools like re2c can convert 100s of regexs and CFG into a single optimized state machine (which includes no back tracking, as far as I remember). It should easily handle 10s of millions transactions per second per core if the complete state machine fits into the CPU level 3 cache (or lower), with a bi…

There is also Ragel [0], but I think that in this context deploying regexes as strings is safer than generating code and deploying that code (unless Ragel could generate webassembly). [0]: http://www.colm.net/open-source/ragel/

Sorry I didn't see the parent you were responding too, so my point is actually the same as you already made. Thanks.

Re: Details of the Cloudflare outage on July 2, 2019

#158

Post by stevens32 » Wed, Jul 17, 2019, 5:38 PM UTC

Buge wrote: post #120

stevens32 wrote: post #85

For the regex novices here, would anyone mind explaining what that pattern is meant to match? More specifically, what `. (?:. =.*)` is meant to do?

It's meant to match any number of any characters, then match an equal sign, then match any number of any characters. But it's very badly written. It should instead simply be written .*=.* BTW, your comment got mangled by HN's markdown formatting.

Gotcha - I thought I was just missing the point as to why it wasn't simpler since it looked to have been structured that way intentionally.

Re: Details of the Cloudflare outage on July 2, 2019

#159

Post by stevens32 » Wed, Jul 17, 2019, 5:39 PM UTC

Rapzid wrote: post #101

stevens32 wrote: post #85

For the regex novices here, would anyone mind explaining what that pattern is meant to match? More specifically, what `. (?:. =.*)` is meant to do?

I could tell you but I also want to tell you to plug that bad boy into https://regex101.com/ . It will give you a written explanation of the regexp on the right. And now, would you have believed I knew without peeking? ;)

That is a super handy site, thanks!