Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on June 21, 2022

blog.cloudflare.com

Post reply on HN
Print view View original

161–170 of 234 posts

Re: Cloudflare outage on June 21, 2022

#161

Post by grenbys »

Would be great if the timeline covered 19 minutes of 6:32 – 06:51. How long did it take to get the right people on the call? How long did it take to identify deployment as a suspect?

Another massive gap is the rollback: 6:58 – 7:42 – 44 minutes! What exactly was going on and why did it take so long? What were those back-up procedures mentioned briefly? Why engineers where stepping on each other toes? What's the story with reverting reverts?

Adding more automation, tests and fixing that specific ordering issue of course is an improvement. But that adds more complexity and any automation ultimately will fail some day.

Technical details are all appreciated. But it is going to be something else next time. Would be great to learn more about human interactions. That's where the resilience of a socio-technical system happened and I bet there is some room for improvement there.

Re: Cloudflare outage on June 21, 2022

#162

Post by systemvoltage »

grenbys wrote: post #161

Would be great if the timeline covered 19 minutes of 6:32 – 06:51. How long did it take to get the right people on the call? How long did it take to identify deployment as a suspect? Another massive gap is the rollback: 6:58 – 7:42 – 44 minutes! What exactly was going on and why did it take so long? What were those back-up procedures mentioned briefly? Why engineers where stepping on each other toes? What's the story…

It would be fun to be a fly on the wall when shit hits the fan in general. From Nuclear meltdowns to 9/11 ATC recordings, it is fascinating to see how emergencies play out and what kind of things go on with boots-on-ground, all-hands-on-deck situations.

Like, does Cloudflare have an emergency procedure for escalation? What does that look like? How does the CTO get woken up in the middle of the night? How to get in touch with critical and most important engineers? Who noticed Cloudflare down first? How do quick decisions get made and decided? Do people get on a giant zoom call? Or emails going around? What if they can't get hold of the most important people that can flip switches? Do they have a control room like the movies? CTO looking over the shoulder calling "Affirmative, apply the fix." followed by a progress bar painfully moving towards completion.

Re: Cloudflare outage on June 21, 2022

#163

Post by prophesi »

jgrahamc wrote: post #127
herpderperator wrote: post #125

Earlier quoted context omitted.

I didn't quite understand this. It sounds like Cloudflare's outage didn't affect you depite being their customer. Why did their large outage not affect you?

It wasn't a global outage.

I thought it was global? 19 data centers were taken offline which "handle a significant proportion of [Cloudflare's] global traffic".

Re: Cloudflare outage on June 21, 2022

#164

Post by ihaveajob »

prophesi wrote: post #163
jgrahamc wrote: post #127

Earlier quoted context omitted.

It wasn't a global outage.

I thought it was global? 19 data centers were taken offline which "handle a significant proportion of [Cloudflare's] global traffic".

But if your clients are mostly asleep while this is happening, they might not notice.

Re: Cloudflare outage on June 21, 2022

#165

Post by ethbr0 »

saghm wrote: post #108
smugma wrote: post #60

Earlier quoted context omitted.

I agree that this is a free ad/recruitment. However, it’s easy to see how more conservative businesses see this as a risk. They are highlighting their deficiencies, letting their big important clients know that human error can bring their network down. Additionally, these post-mittens work for Cloudflare because they have a great reputation and good uptime. If this were happening daily or weekly, it would be a warnin…

OTOH, I think most actual engineers would know that everywhere has deficiencies and can be brought down by human error, and I'd personally rather use a product where the people running it admit this rather than just claim that their genius engineers made it 100% foolproof and nothing could ever possibly go wrong

Absolutely. The first step of good SRE is admitting (publicly and within the organization) that you have a problem.

Re: Cloudflare outage on June 21, 2022

#166

Post by wstuartcl »

yuliyp wrote: post #137
gouggoug wrote: post #126

Earlier quoted context omitted.

I'd be super interested in understanding what this means concretely. For example, are we talking about reverting commits? If so, why were engineers reverting reverts?

Developer 1 fetches code, changes flag A. Rebuilds config. Developer 2 fetches code, changes flag B. Rebuilds config. Developer 1 deploys built config. Developer 2 deploys built config, inadvertently reverts developer 1's changes.

also can happen when your deploy process has two flows for revert a forward movement revert (where new bits and head are committed fixing the items that needed to be reverted) and a "previous head" revert which just goes back one revision in the rcs (or tagged version).

Imagine the first eng team did a forward movement revert that corrected the issue and had a new head bits that gets deployed, where shortly after another eng fires off the second process type and tells the system to pull back to the last revision (which is now the bad revision as it was just replaced with fresher deploy bits).

Having two revert processes in the toolkit and maybe a few disperse teams working to revert the issue without tight communication leads to this issue.

I think this is more likely the basis issue vs a bad merge (I assume that the root cause was broadcasted wide and large to anyone making a merge)

Re: Cloudflare outage on June 21, 2022

#167

Post by ethbr0 »

zaidf wrote: post #145
viraptor wrote: post #19

Earlier quoted context omitted.

I feel like others lose opportunities by not doing the same. By publishing early and publishing the details they: keep the company in the news with positive stuff (free ad), get an internal documentation of the incident (ignoring the customer oriented "we're sorry" part), effectively get a free recruitment post (you're reading this because you're in tech and we do cool stuff, wink), release some internal architecture…

>I feel like others lose opportunities by not doing the same IMO it is a slippery slope to see this as opportunity too strongly. Sure, doing the right thing may be net beneficial to the business in the long run...but the $RIGHT_THING should be done first and foremost because it's the right thing.

I believe Marcus Aurelius had something similar to say on the matter. :-)

Re: Cloudflare outage on June 21, 2022

#168

Post by jgrahamc »

prophesi wrote: post #163
jgrahamc wrote: post #127

Earlier quoted context omitted.

It wasn't a global outage.

I thought it was global? 19 data centers were taken offline which "handle a significant proportion of [Cloudflare's] global traffic".

I am in Lisbon and was not having trouble because Cloudflare's Lisbon data center was not affected. But over in Madrid there was trouble. It depended where you are.

Re: Cloudflare outage on June 21, 2022

#169

Post by homero »

psim1 wrote: post #104

CF is the only company I have ever seen that can have an outage and get pages of praise for it. I don't have any (current) use for CloudFlare's products but I would love to see the culture that makes them praiseworthy spread to other companies.

I'm also a huge fan

Re: Cloudflare outage on June 21, 2022

#170

Post by mkl »

prophesi wrote: post #163
jgrahamc wrote: post #127

Earlier quoted context omitted.

It wasn't a global outage.

I thought it was global? 19 data centers were taken offline which "handle a significant proportion of [Cloudflare's] global traffic".

From the article: "Depending on your location in the world you may have been unable to access websites and services that rely on Cloudflare. In other locations, Cloudflare continued to operate normally."
Post reply on HN