Live data from Hacker News

Cloudflare outage on June 21, 2022

blog.cloudflare.com

121–130 of 234 posts

Re: Cloudflare outage on June 21, 2022

#121

BGP changes should be like the display resolution changes on your PC... It should revert as a failsafe if not confirmed within X minutes.

That's the "commit-confirm" process they mention they will use in the write-up: > Primarily, we will be concentrating on automation improvements ... and provide an automated “commit-confirm” rollback.

Surprised everyone has not switched to this already - great idea

Re: Cloudflare outage on June 21, 2022

#122

Time and time again, this type of response proves that it's the right way handle a bad situation. Be humble, apologize, own your mistake, and give a transparent snapshot into what went wrong and how you're going to learn from the mistake. Or you could go the opposite direction and risk turning something like this into a PR death spiral.

Exactly. I trust businesses/people that are transparent about their mistakes/failures much more than the ones that avoid them (except Apple which never accepts their mistakes, but I still trust their products, I think I'm affected by RDF). At the end of the day, everybody makes mistakes and that's okay. Everybody else also know that everybody makes mistakes. So why not accept it? I really don't get what's wrong with…

The exception that proves the rule with Apple:

https://appleinsider.com/articles/12/09/28/apple-ceo-tim-coo...

Re: Cloudflare outage on June 21, 2022

#123

Really interesting that 19 cities handle 50% of the requests.

Actually, I think the flip side is even more interesting. If you want to give good, low latency service to 50% of the world you need a lot of data centers.

If you have an efficient website, you can get decent performance to most of the world with one pop on the West cost of the USA.

Re: Cloudflare outage on June 21, 2022

#124
post #46

It's nearly always BGP when this level of failure occurs.

I dunno man, you can really fuck things up with DNS also.

I was on a severely understaffed edge team fronting several thousand engineers at a fortune 500 - every deploy felt like a spacex launch from my cubicle. I have a lot of reverence for the engineers who take on that kind of responsibility.

Re: Cloudflare outage on June 21, 2022

#125

One of our sites uses Cloudflare and serves 400k pageviews per month and generates around $650/day in ad and affiliate revenue. If the site is not up the business is not making any money. Looking at the hourly chart in Google Analytics (compared to the previous day) there isn't even a blip during this outage. So for all the advantages we get from Cloudflare (caching, WAF, security [our WP admin is secured with Cloudf…

I didn't quite understand this. It sounds like Cloudflare's outage didn't affect you depite being their customer. Why did their large outage not affect you?

Re: Cloudflare outage on June 21, 2022

#126
post #5

07:42: The last of the reverts has been completed. This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. Ouch

I'd be super interested in understanding what this means concretely. For example, are we talking about reverting commits? If so, why were engineers reverting reverts?

Re: Cloudflare outage on June 21, 2022

#127

One of our sites uses Cloudflare and serves 400k pageviews per month and generates around $650/day in ad and affiliate revenue. If the site is not up the business is not making any money. Looking at the hourly chart in Google Analytics (compared to the previous day) there isn't even a blip during this outage. So for all the advantages we get from Cloudflare (caching, WAF, security [our WP admin is secured with Cloudf…

I didn't quite understand this. It sounds like Cloudflare's outage didn't affect you depite being their customer. Why did their large outage not affect you?

It wasn't a global outage.

Re: Cloudflare outage on June 21, 2022

#128

One of our sites uses Cloudflare and serves 400k pageviews per month and generates around $650/day in ad and affiliate revenue. If the site is not up the business is not making any money. Looking at the hourly chart in Google Analytics (compared to the previous day) there isn't even a blip during this outage. So for all the advantages we get from Cloudflare (caching, WAF, security [our WP admin is secured with Cloudf…

I didn't quite understand this. It sounds like Cloudflare's outage didn't affect you depite being their customer. Why did their large outage not affect you?

Because of the time at which the outage occurred, most of this person's customers were not trying to access the site.
Post reply on HN