Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Details of the Cloudflare outage on July 2, 2019

blog.cloudflare.com

Post reply on HN
Print view View original

71–80 of 159 posts

Re: Details of the Cloudflare outage on July 2, 2019

#71

Post by Twirrim »

One thing set of alarm bells in my head from an operational perspective:

> Switching to either the re2 or Rust regex engine which both have run-time guarantees. (ETA: July 31)

That's short timescales for quite a significant change. I know it's just replacing a piece of automation with one that does the same task, but the guts are all changing and all automation introduces some level of instability, and a bunch of unknowns. Changing the regex engine is just as significant as introducing new automation from an operations perspective, even if it seems like it should be a no-brainer. I'd encourage taking time there (unless this is something they've been working on a lot and are already doing canary testing).

The other steps look excellent, and they should all collectively give ample breathing room to make sure that switching to re2 or Rust's regex engine won't introduce further issues. There's no need to be doing it on a scale of weeks.

Some quick thoughts about Quicksilver: Deploying everywhere super fast is inherently dangerous (for some reason, old school rocketjumping springs to mind. Fine until you get it wrong).

I definitely see the value for customer actions, but for WAF rule rollouts, some kind of (automated) increasing speed rollouts might be good, and might help catch issues even as the deployment steps beyond the bounds of PIG etc. canary fleets. Of course, that's also useless in and of itself unless there is some kind of automated feedback mechanism to retard, stop, or undo changes.

If I can make a reading suggestion: https://smile.amazon.com/gp/product/0804759464/ref=ppx_yo_dt... The book is "High Reliability Management: Operating on the Edge (High Reliability and Crisis Management)" (unfortunately not available in electronic form). It's focussed on the energy grid in California, the authors were university researchers specialising in high reliability operations, and they had the good fortune to be present doing a research job at the operations centre right when the California brownouts were occurring in the early 2000s. There's a lot to be gleaned from that book, particularly when it comes to automation, and especially changes to automation.

Re: Details of the Cloudflare outage on July 2, 2019

#72

Post by dsl »

buildzr wrote: post #24

> Then we moved on to restoring the WAF functionality. Because of the sensitivity of the situation we performed both negative tests (asking ourselves “was it really that particular change that caused the problem?”) and positive tests (verifying the rollback worked) in a single city using a subset of traffic after removing our paying customers’ traffic from that location. Haha, so the free customers are crash test dum…

> I'd much rather "be the product" in this way than in the way ad companies cause at least

Your customers are the product. Cloudflare sets a first party tracking cookie on every domain they serve. They unwrap TLS and can see every product your customers look at or buy.

Whether intentionally or not, they built the Ad Network 2.0. They found the solution to ISPs not being able to snoop, and browsers locking down third party tracking.

Re: Details of the Cloudflare outage on July 2, 2019

#74

Post by xxyeah »

dang wrote: post #70
_wmd wrote: post #13

So in response to a catastrophic failure due to testing in prod, they're going to push out a brand new regex engine with an ETA of 2 weeks. Can anyone say testing in prod? The constant use of 'I' and 'me' (19 occurrences in total) deeply tarnishes this report, and repeatedly singling out a responsible engineer, nameless or not, is a failure in its own right. This was a collective failure, any individual identity is t…

Personal attacks aren't allowed on HN, and please don't post in the flamewar style here generally. https://news.ycombinator.com/newsguidelines.html

You are overreacting and protecting your preferred people. What is HN running on again?

If this is a personal attack, there are literally 10-50 of these per day in arbitrary threads.

Re: Details of the Cloudflare outage on July 2, 2019

#75

Post by [deleted] »

toomuchtodo wrote: post #4
woliveirajr wrote: post #2

> Unfortunately, last Tuesday’s update contained a regular expression that backtracked enormously and exhausted CPU used for HTTP/HTTPS serving. One of those cases where they had 1 problem, used regular expression and ended up with 2 problems ? Edit: I really like how much information is given by CloudFlare. 11 points in the "what went wrong analysis" is how every root-cause analysis should be done.

Somewhat humorous, as someone [1] (congrats /u/fossuser!) mentioned this failure scenario in the thread about Twitter being down yesterday. "Pushing bad regex to production, chaos monkey code causing cascading network failure, etc.", in response to a comment from someone who previously worked at Cloudflare. [1] https://news.ycombinator.com/item?id=20415608

[deleted]

Re: Details of the Cloudflare outage on July 2, 2019

#76

Post by londons_explore »

Cloudflare lets their customers write their own WAF regex rules right?

And those rules still get run on every box on cloudflares edge network with HTTP requests from strangers on the internet right?

So how come this didn't get triggered by a customer first?

Perhaps it did get triggered by a customer first, but that customer didn't get too much traffic of the URL which triggers the issue, and that box got one thread stuck executing that regex for a few minutes till a health check killed it...? Does this imply that cloudflare runs with random failing health checks across the fleet and there isn't someone looking at core dumps of such failures?

That would align with my experience with seeing occasional "502 bad gateway" errors from cloudflare over the past few years. It also seems likely considering the incident where cloudflare servers leaked sensitive memory contents into HTTP responses which happened so frequently they got cached by google search. Hard to leak arbitrary memory contents without occasional SIGSEGV's...

If the above conjecture is true, it reflects very badly on engineering culture at Cloudflare. The core issue had been seen across the fleet sporadically for a long time, but was ignored, and even during the postmortem process, which should be a very thorough investigation, the telltale pre-warning signs of the issue were still missed.

Re: Details of the Cloudflare outage on July 2, 2019

#77

Post by edwintorok »

lkoolma wrote: post #61

Might be late, but has anyone in CloudFlare tried to switch away from regex to something more efficient and powerful? Tools like re2c can convert 100s of regexs and CFG into a single optimized state machine (which includes no back tracking, as far as I remember). It should easily handle 10s of millions transactions per second per core if the complete state machine fits into the CPU level 3 cache (or lower), with a bi…

There is also Ragel [0], but I think that in this context deploying regexes as strings is safer than generating code and deploying that code (unless Ragel could generate webassembly).

[0]: http://www.colm.net/open-source/ragel/

Re: Details of the Cloudflare outage on July 2, 2019

#78

Post by eastdakota »

bostik wrote: post #60
kevinherron wrote: post #48

Sometimes I wish Cloudflare failed more often so I could read more of these postmortems...

Please no. The last time they had a global problem, everyone scrambled for more than a week. (Cloudbleed) This 30-minute global outage was pretty nasty, but not anywhere near as awful. Timing helped, as nothing truly critical was affected. (There are some extremely high-volume sporting events which, if affected even just for few minutes, can have a direct impact on the bottom line.) I do not wish to see more of these…

Agree. Cloudbleed was really, really awful. We should write up all the things we learned from that and all the changes we've made since. Just looking at the number of engineers who are Rust experts since then, for instance.

Re: Details of the Cloudflare outage on July 2, 2019

#79

Post by dang »

xxyeah wrote: post #74
dang wrote: post #70

Earlier quoted context omitted.

Personal attacks aren't allowed on HN, and please don't post in the flamewar style here generally. https://news.ycombinator.com/newsguidelines.html

You are overreacting and protecting your preferred people. What is HN running on again? If this is a personal attack, there are literally 10-50 of these per day in arbitrary threads.

That comment was breaking the site guidelines, quite badly in fact. We moderate comments like that the same way regardless of who or what they're about.

> there are literally 10-50 of these per day in arbitrary threads

If you can find cases of this where moderators didn't respond, I'd like to see links. The likeliest explanation is simply that we didn't see it. We don't come close to seeing everything that gets posted here, so we depend on users, via flagging (https://news.ycombinator.com/newsfaq.html) or by emailing hn@ycombinator.com.

> What is HN running on again?

I suppose I have to answer this or someone will concoct a sinister reason why I didn't. HN doesn't run on Cloudflare.

Re: Details of the Cloudflare outage on July 2, 2019

#80

Post by hinkley »

w0m wrote: post #62

>A protection that would have helped prevent excessive CPU use by a regular expression was removed by mistake during a refactoring of the WAF weeks prior—a refactoring that was part of making the WAF use less CPU. Faster karma than normal i think.

Taking the safeties off to go faster... yes, you will go faster, but it might be right off a cliff.

This is a good lesson on Chesterton’s Fence. I’ve been thinking for a while that we really need the (default behavior) ability to annotate commits after the fact, so that we have a durable commentary that can evolve over time. We should be able to go back and add strongly worded things like “yes this looks broken but it exists due to this bug fix” or “please don’t write new code that looks like this. See xyz for a better alternative.”

Hell I think I’d be perfectly ok if the code review lived with the code permanently. Regression in the code? Josh warned you it was a bad idea. Maybe we should listen to Josh more?

Post reply on HN