Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on June 21, 2022

blog.cloudflare.com

Post reply on HN
Print view View original

41–50 of 234 posts

Re: Cloudflare outage on June 21, 2022

#41

Post by kevin_nisbet »

samwillis wrote: post #2

In a world where it can take weeks for other companies to publish a postmortem after an outage (if they ever do), I never ceases to amaze me how quickly CF manage to get something like this out. I think it's a testament to their Ops/Incident response teams and internal processes, it builds confidence in their ability to respond quickly when something does go wrong. Incredible work!

I agree, I think the transparency builds trust and I encourage it where I can. The counter thought I had when reading this case though, is it almost feels too fast. What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done.

Doing incident response (both outage and security), the tactical fixes for a specific problem are usually pretty easy. We can fix a bug, or change this specific plan to avoid the problem. The search for conditions that allowed the incident to occur can be alot more time consuming, and most organizations I've worked for are happy to make a couple tactical changes and move on.

Re: Cloudflare outage on June 21, 2022

#42

Post by alanning »

rubatuga wrote: post #38

Uh, shouldn’t there be a staging environment for these sort of changes?

Yes, that was one of the issues they mentioned in the post. Not that they didn’t have a staging/testing environment but that it didn’t include the specific type of new architecture configuration, “MCP”, that ultimately failed.

One of their future changes is to include MCPs in their testing environments.

Re: Cloudflare outage on June 21, 2022

#43

Post by samwillis »

johnklos wrote: post #30
samwillis wrote: post #7

Earlier quoted context omitted.

CloudFlare are a hosting provider and CDN, they aren't "push[ing] ... hard for recentralization of the web". If it was AWS, Akamai, Google Cloud, or any of the other massive providers this comment wouldn't be made. I don't really understand the association between centralisation and CloudFlare, other than it being a Meme.

I think you've already drunk the Flavor Aid. What do you have when you have all DNS going through them, via DoH, and all web requests going through them, if not recentralization? Sure, they want us to think they give us the freedom to host our web sites anywhere because they're "protected" by them, but that "protection" means we've agreed to recentralize. It's pretty dismissive to describe something as a meme just be…

I understand that for their WAF, DDOS and threat detection products they need to have a very large amount of traffic going through them. They have been very aggressive with their free service to achieve that, to the benefit of all their customers (including the free ones). Some could see that as a push to at centralisation, I don't.

What I don't understand, or believe, is that they want to be the sole (as in centralised) network for the internet. I don't believe they as a company, or the people running it, want that. They obviously have ambition to be one of the largest networking/cloud providers, and are achieving that.

I don't intend either to dismiss your concerns (which are a legitimate thing to have, centralisation would be very bad), my suggestion with the meme comment is that there is at times a trend to "brigade" on large successful companies in a meme-like way. That isn't to suggest you were.

Re: Cloudflare outage on June 21, 2022

#44

Post by rocky_raccoon »

can16358p wrote: post #36
rocky_raccoon wrote: post #8

Time and time again, this type of response proves that it's the right way handle a bad situation. Be humble, apologize, own your mistake, and give a transparent snapshot into what went wrong and how you're going to learn from the mistake. Or you could go the opposite direction and risk turning something like this into a PR death spiral.

Exactly. I trust businesses/people that are transparent about their mistakes/failures much more than the ones that avoid them (except Apple which never accepts their mistakes, but I still trust their products, I think I'm affected by RDF). At the end of the day, everybody makes mistakes and that's okay. Everybody else also know that everybody makes mistakes. So why not accept it? I really don't get what's wrong with…

> I really don't get what's wrong with accepting mistakes, learning from them, and moving on.

Some people really struggle with this (myself included) but I think it's one of the easiest "power ups" you can use in business and in life. The key is that you have to actually follow through on the "learning from them" clause.

Re: Cloudflare outage on June 21, 2022

#45

Post by jgrahamc »

kevin_nisbet wrote: post #41
samwillis wrote: post #2

In a world where it can take weeks for other companies to publish a postmortem after an outage (if they ever do), I never ceases to amaze me how quickly CF manage to get something like this out. I think it's a testament to their Ops/Incident response teams and internal processes, it builds confidence in their ability to respond quickly when something does go wrong. Incredible work!

I agree, I think the transparency builds trust and I encourage it where I can. The counter thought I had when reading this case though, is it almost feels too fast. What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done. Doing incident response (both outage and security), the tactical fixes for a specific problem are usual…

What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done.

There is not. From here there's an ongoing process with a formal post-mortem, all sorts of tickets tracking work to prevent further reoccurrence. This post is just the beginning internally.

Re: Cloudflare outage on June 21, 2022

#50

Post by minecraftchest1 »

Something else that I think would be smart to implement is a reorder detection. Have the change approval specificy point out stuff that gets reordered, and require manual approval for each section that gets moved around.

I also think that having a script that walks through the file and points out any ovibious mistakes would be good to have as well.

Post reply on HN