Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage caused by bad software deploy

blog.cloudflare.com

Post reply on HN
Print view View original

111–120 of 137 posts

Re: Cloudflare outage caused by bad software deploy

#111

Post by runeks »

tehlike wrote: post #99
aleem wrote: post #55

If a single regex can take down the Internet for a half hour, that's definitely not good -- for a class of errors that can be easily prevented, tested, etc. The timing is unfortunate too, after calling out Verizon for lack of due process and negligence. I'm sure they have an undo or rollback for deployments but probably worth investing into further. They also need to resolve the catch-22 where people could not login…

It should be taken as a given that testing is necessary but not sufficient to prevent production outages, or limit their impact. Monitoring, canaries, experimantations do need to be adopted at pretty much everywhere possible.

> It should be taken as a given that testing is necessary but not sufficient to prevent production outages […]

That depends on how good your tests are.

Re: Cloudflare outage caused by bad software deploy

#112

Post by lima »

pfundstein wrote: post #69
aleem wrote: post #55

If a single regex can take down the Internet for a half hour, that's definitely not good -- for a class of errors that can be easily prevented, tested, etc. The timing is unfortunate too, after calling out Verizon for lack of due process and negligence. I'm sure they have an undo or rollback for deployments but probably worth investing into further. They also need to resolve the catch-22 where people could not login…

> The timing is unfortunate too, after calling out Verizon for lack of due process and negligence. Nonetheless, Verizon could take a leaf out of their responsiveness and transparency book.

Yeah. They criticized Verizon for being unresponsive. Mistakes happen.

Re: Cloudflare outage caused by bad software deploy

#113

Post by ohyeshedid »

JaimeThompson wrote: post #35
Lorin wrote: post #18

Earlier quoted context omitted.

Deploy once, never update, and deploy a missile defense to prevent backhoes from digging up fiber?

You honestly think a missile defense system will work. Backhoes are much more creative than that. You will need defense in depth, roaming patrols, as well as air and satellite based monitoring assets.

https://i.imgur.com/rDW7W3d.png

Re: Cloudflare outage caused by bad software deploy

#114

Post by londons_explore »

ti_ranger wrote: post #105

> We make software deployments constantly across the network and have automated systems to run test suites and a procedure for deploying progressively to prevent incidents. Good. > Unfortunately, these WAF rules were deployed globally in one go and caused today’s outage. Wow. This seems like a very immature operational stance. Any deployment of any kind should be subject to minimum deployment safety, that they claim…

So many employees deploying so many changes at a time it wasn't clear which one was the cause...?

Re: Cloudflare outage caused by bad software deploy

#115

Post by mschuster91 »

londons_explore wrote: post #114
ti_ranger wrote: post #105

> We make software deployments constantly across the network and have automated systems to run test suites and a procedure for deploying progressively to prevent incidents. Good. > Unfortunately, these WAF rules were deployed globally in one go and caused today’s outage. Wow. This seems like a very immature operational stance. Any deployment of any kind should be subject to minimum deployment safety, that they claim…

So many employees deploying so many changes at a time it wasn't clear which one was the cause...?

Which is why the entire (mostly in Agile environments) model of "deploy to prod as soon as you can" is absolute nuts.

If you're dev at a hipster app maybe a dozen people use to holler "yo" at each other, by all means go for it. If you're operating one of the biggest and most important chonks of Internet infra... maaaaaybe stick to established practices such as stage testing, release schedules and incremental rollouts?

Re: Cloudflare outage caused by bad software deploy

#116

Post by mschuster91 »

thegagne wrote: post #109
txcwpalpha wrote: post #72

Earlier quoted context omitted.

AFAIK Cloudflare isn't in any way a "customer" of Verizon. Verizon doesn't owe Cloudflare any kind of response or devotion of resources. Verizon owes it's actual customers a resolution to their problem, which they gave . I'm not saying Verizon is perfect nor absolved of fault, but Cloudflare was/is not owed any kind of explanation or assistance by VZ, and it's absurd of CF to still be whining about that fact (as they…

I’d say what they really need is a representative governing body over major network carriers to establish proper standards and levy fines for those that do not comply. Kind of similar to a homes association saying “hey that trash on your lawn affects your neighbor, clean it up!” It’s true that they are not a customer but at that level what they do affects each other, and it’s better to resolve things civilly and priv…

In theory this should be the job of the FCC, and in Europe the local regulatory agencies (BNetzA in Germany for example). But properly funding them to do their jobs doesn't seem to be very high om the political agendas these days.

Re: Cloudflare outage caused by bad software deploy

#117

Post by nightfly »

rainyMammoth wrote: post #78
_wmd wrote: post #73

Earlier quoted context omitted.

You'd think after leaking private data for literally months less than 3 years ago (and only noticing because Google had to point it out to them) that they'd, y'know, have at least some kind of QA environment fed with sample traffic by now. Really hard to believe they're still getting caught testing in prod

For working in that field, the arrogance of CloudFlare is still unbelievable to me. After their huge Cloudbleed issue with the addition of this one, they continue to call out everyone through their blog posts. And everyone seems fine with it because they are a hype company.

As a random outsider who really couldn't care less about the service CloudFlare provides: their responses to outages and transparency is really great and I wish more tech companies would do the same. It gets tiring hearing about large outages at over services/providers and only learning that they were caused by "network partitions", or other networking issues. Every company has to deal with these issues and CloudFlare does an awesome job at letting me at least learn something about what went wrong when these incidents happen.

Re: Cloudflare outage caused by bad software deploy

#118

Post by martin_a »

aleem wrote: post #55

If a single regex can take down the Internet for a half hour, that's definitely not good -- for a class of errors that can be easily prevented, tested, etc. The timing is unfortunate too, after calling out Verizon for lack of due process and negligence. I'm sure they have an undo or rollback for deployments but probably worth investing into further. They also need to resolve the catch-22 where people could not login…

> If a single regex can take down the Internet for a half hour, that's definitely not good

As I commented yesterday, this is due to the fact, that "the Internet" thinks it needs to use Cloudflare services, although there really is no need to do so.

Stupid people making stupid decisions and then wondering why their services are down.

Re: Cloudflare outage caused by bad software deploy

#119

Post by flurdy »

mschuster91 wrote: post #115
londons_explore wrote: post #114

Earlier quoted context omitted.

So many employees deploying so many changes at a time it wasn't clear which one was the cause...?

Which is why the entire (mostly in Agile environments) model of "deploy to prod as soon as you can" is absolute nuts. If you're dev at a hipster app maybe a dozen people use to holler "yo" at each other, by all means go for it. If you're operating one of the biggest and most important chonks of Internet infra... maaaaaybe stick to established practices such as stage testing, release schedules and incremental rollouts…

Why not both.

I don't want to return to the old slothful release schedules of the 2000s, where features and bug fixes was mostly stagnant.

You can have staging, scheduled QA signed off releases, that happen every day. I have worked on some fairly large significant services and we still released several times a day, just that you did not trigger the final prod release yourself but the QAs pressed the button instead. Though usually just once a day per microservice.

I have also worked with several clients lately without QA where devs could themselves push to prod many times a day. I am not sure these systems were that much less stable, though they were all mostly greenfield and not critical public government systems. They were off course a lot smaller changes, and quick to undo. Which is the core element of "release straight to prod" ethos.

I am sure Cloudflare have a significant QA process whilst using todays fast moving release schedules.

What is always a grey-zone is configuration changes. Even if properly versioned and on a release schedule train with several staging environments, configuration is often very environment sensitive. So maybe they could not test it properly in any staging environments but had to hope prod worked...

However Cloudflare will hopefully implement some way to make sure this particular configuration and subsequent future changes are not as bottle-necked that instead can be be gradually released to a subset and region-by-region instead of a big bang to all. Though canary/blue-green/etc releases of core routing configurations is hard.

Re: Cloudflare outage caused by bad software deploy

#120

Post by issati »

rainyMammoth wrote: post #78
_wmd wrote: post #73

Earlier quoted context omitted.

You'd think after leaking private data for literally months less than 3 years ago (and only noticing because Google had to point it out to them) that they'd, y'know, have at least some kind of QA environment fed with sample traffic by now. Really hard to believe they're still getting caught testing in prod

For working in that field, the arrogance of CloudFlare is still unbelievable to me. After their huge Cloudbleed issue with the addition of this one, they continue to call out everyone through their blog posts. And everyone seems fine with it because they are a hype company.

I don't use CloudFlare nor have any interest in them, but I don't see the arrogance. The issues CloudFlare have are things everyone takes seriously and are working very hard on. Deployment and memory safety are hard problems that happens to the best of the best. It happens Google, Amazon and Facebook. If anything the idea that this would damaging, because it is more public, is arrogant. If CloudFlare would be saying that everything is fine you might have a point, but they aren't. Just like the other companies mentioned they seem to be improving their routines, programming and infrastructure to try and mitigate these problems.

What they are criticising however are things like not adopting new protocols or not taking things that affects everyone seriously. This isn't something that would happen if people were trying. And the response from some of the industry is "we know what we are doing", and shortly after the same thing happens again and again and again.

So I don't really see CloudFlare being that arrogant, if anything it's the "you are not better than us" from some parts of the industry that is. The day I see CloudFlare not trying I would be happy calling them arrogant. But if anything I would caution that they are too successful by trying more than most.

Post reply on HN