Cloudflare outage caused by bad software deploy

Post by pornel » Wed, Jul 03, 2019, 10:04 AM UTC

londons_explore wrote: post #114

ti_ranger wrote: post #105

> We make software deployments constantly across the network and have automated systems to run test suites and a procedure for deploying progressively to prevent incidents. Good. > Unfortunately, these WAF rules were deployed globally in one go and caused today’s outage. Wow. This seems like a very immature operational stance. Any deployment of any kind should be subject to minimum deployment safety, that they claim…

So many employees deploying so many changes at a time it wasn't clear which one was the cause...?

Post by jaclaz » Wed, Jul 03, 2019, 11:35 AM UTC

klodolph wrote: post #13

eganist wrote: post #6

Kinda wonder at this point what findings exist on their Availability SOC 2, assuming they've gotten one. The repeated outages plus the constant malicious advertising by scammy ad providers through cloudflare are slowly turning me off to the service as a potential enterprise customer. Unfortunate too since plenty of superlatively qualified people build great things there (hat tip to Nick Sullivan), but it seems like t…

As an engineer, I get pissed whenever I see 100% uptime, or eleven-nines, nine-nines, or other impossible targets. Like, how am I supposed to design a system with numbers like that?

Post by kostko » Wed, Jul 03, 2019, 1:16 PM UTC

rainyMammoth wrote: post #78

_wmd wrote: post #73

Earlier quoted context omitted.

You'd think after leaking private data for literally months less than 3 years ago (and only noticing because Google had to point it out to them) that they'd, y'know, have at least some kind of QA environment fed with sample traffic by now. Really hard to believe they're still getting caught testing in prod

For working in that field, the arrogance of CloudFlare is still unbelievable to me. After their huge Cloudbleed issue with the addition of this one, they continue to call out everyone through their blog posts. And everyone seems fine with it because they are a hype company.

Post by PopeDotNinja » Wed, Jul 03, 2019, 1:56 PM UTC

lanrh1836 wrote: post #81

neom wrote: post #15

I'm always beyond impressed with how responsive and transparent CF is with incidence and post mortem communication. Given who the CEO and COO are, I suppose this shouldn't be surprising, never the less as a customer it builds a great deal of trust. Kudos.

A quick look at their Glassdoor reviews paints a very different story if the reviews are to be believed...

Post by ReidZB » Wed, Jul 03, 2019, 2:56 PM UTC

syntheticcdo wrote: post #103

ReidZB wrote: post #95

Earlier quoted context omitted.

AWS services have their own individual SLAs. Route53, in particular, has a 100% SLA: https://aws.amazon.com/route53/sla/ (To my knowledge, it's the only AWS service to promise 100%.)

Interesting distinction here: 100% SLA on responding to incoming DNS requests. The R53 console or management interfaces could be down and the SLA stays in tact-- if you can't update your DNS then 100% incorrect responses isn't very helpful.