Cloudflare outage on June 21, 2022 • Hacker News Board

Re: Cloudflare outage on June 21, 2022

#131

Post by dylan604 » Tue, Jun 21, 2022, 5:26 PM UTC

coob wrote: post #122

can16358p wrote: post #36

Earlier quoted context omitted.

Exactly. I trust businesses/people that are transparent about their mistakes/failures much more than the ones that avoid them (except Apple which never accepts their mistakes, but I still trust their products, I think I'm affected by RDF). At the end of the day, everybody makes mistakes and that's okay. Everybody else also know that everybody makes mistakes. So why not accept it? I really don't get what's wrong with…

The exception that proves the rule with Apple: https://appleinsider.com/articles/12/09/28/apple-ceo-tim-coo...

"Is it Apple Maps bad?" --Gavin Belson, Silicon Valley

This one line will forever cement exactly how bad Apple Maps' release was. Thanks Mike Judge!

Re: Cloudflare outage on June 21, 2022

#132

Post by dylan604 » Tue, Jun 21, 2022, 5:27 PM UTC

rocky_raccoon wrote: post #44

can16358p wrote: post #36

Earlier quoted context omitted.

Exactly. I trust businesses/people that are transparent about their mistakes/failures much more than the ones that avoid them (except Apple which never accepts their mistakes, but I still trust their products, I think I'm affected by RDF). At the end of the day, everybody makes mistakes and that's okay. Everybody else also know that everybody makes mistakes. So why not accept it? I really don't get what's wrong with…

> I really don't get what's wrong with accepting mistakes, learning from them, and moving on. Some people really struggle with this (myself included) but I think it's one of the easiest "power ups" you can use in business and in life. The key is that you have to actually follow through on the "learning from them" clause.

Sure, this can be a good thing when it's a rare occurrence. If it is a weekly event, then you just start to look incompetent

Re: Cloudflare outage on June 21, 2022

#133

Post by philipwhiuk » Tue, Jun 21, 2022, 5:28 PM UTC

Is there no system to unit test a rule-set?

Re: Cloudflare outage on June 21, 2022

#134

Post by bpye » Tue, Jun 21, 2022, 5:31 PM UTC

ThaDood wrote: post #87

thesuitonym wrote: post #82

Where does one even start with learning BGP? It always seemed super interesting to me, but not really something that could be dealt with on a small scale, lab type basis. Or am I wrong there?

DN42 https://dn42.eu/Home > gets mentioned a lot. Its basically a big dynamic VPN that you can do BGP stuff with. Pretty cool but I could never get my node working properly.

I started setting that up and totally forgot, maybe I should actually try and peer with someone.

Re: Cloudflare outage on June 21, 2022

#135

Post by star-glider » Tue, Jun 21, 2022, 5:36 PM UTC

edf13 wrote: post #46

It's nearly always BGP when this level of failure occurs.

Generally speaking:

You broke half the internet: BGP You broke half of your company's ability to access the internet: DNS

Re: Cloudflare outage on June 21, 2022

#136

Post by can16358p » Tue, Jun 21, 2022, 5:39 PM UTC

coob wrote: post #122

can16358p wrote: post #36

Earlier quoted context omitted.

Exactly. I trust businesses/people that are transparent about their mistakes/failures much more than the ones that avoid them (except Apple which never accepts their mistakes, but I still trust their products, I think I'm affected by RDF). At the end of the day, everybody makes mistakes and that's okay. Everybody else also know that everybody makes mistakes. So why not accept it? I really don't get what's wrong with…

The exception that proves the rule with Apple: https://appleinsider.com/articles/12/09/28/apple-ceo-tim-coo...

Yeah. Forgot that one. When it first came out it was terrible.

Apparently so terrible that Apple apologized, perhaps for the first (and last) time for something.

Re: Cloudflare outage on June 21, 2022

#137

Post by yuliyp » Tue, Jun 21, 2022, 5:39 PM UTC

gouggoug wrote: post #126

thejosh wrote: post #5

07:42: The last of the reverts has been completed. This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. Ouch

I'd be super interested in understanding what this means concretely. For example, are we talking about reverting commits? If so, why were engineers reverting reverts?

Developer 1 fetches code, changes flag A. Rebuilds config. Developer 2 fetches code, changes flag B. Rebuilds config. Developer 1 deploys built config. Developer 2 deploys built config, inadvertently reverts developer 1's changes.

Re: Cloudflare outage on June 21, 2022

#138

Post by can16358p » Tue, Jun 21, 2022, 5:40 PM UTC

dylan604 wrote: post #131

coob wrote: post #122

Earlier quoted context omitted.

The exception that proves the rule with Apple: https://appleinsider.com/articles/12/09/28/apple-ceo-tim-coo...

"Is it Apple Maps bad?" --Gavin Belson, Silicon Valley This one line will forever cement exactly how bad Apple Maps' release was. Thanks Mike Judge!

Yup. Just remember the episode. IIRC in that context Apple Maps was placed even worse than Windows Vista.

Re: Cloudflare outage on June 21, 2022

#139

Post by dylan604 » Tue, Jun 21, 2022, 5:44 PM UTC

can16358p wrote: post #138

dylan604 wrote: post #131

Earlier quoted context omitted.

"Is it Apple Maps bad?" --Gavin Belson, Silicon Valley This one line will forever cement exactly how bad Apple Maps' release was. Thanks Mike Judge!

Yup. Just remember the episode. IIRC in that context Apple Maps was placed even worse than Windows Vista.

I would agree with that. Apple Maps was worse than the hockey puck mouse or the trashcan macpro. trying to decide if it is worse than the butterfly keyboard, but I think the keyboard wins for the shear fact that it impacted me in a way that was uncorrectable where I could just use a different Maps app

Re: Cloudflare outage on June 21, 2022

#140

Post by scottlamb » Tue, Jun 21, 2022, 5:49 PM UTC

michaelmior wrote: post #53

thejosh wrote: post #5

07:42: The last of the reverts has been completed. This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. Ouch

This was something I was surprised not to see directly addressed in terms of follow up steps. When discussing process changes, they mention additional testing, but nothing to address what seems to be a significant communication gap.

I'm sure they have a more detailed internal postmortem, and I imagine it'd go into that. This is a nice high-level overview. They probably don't want to bury that under details of their communication processes, much less go into exactly who did what when for wide consumption by an audience that may not be on board with blameless postmortem culture.