Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on June 21, 2022

blog.cloudflare.com

Post reply on HN
Print view View original

201–210 of 234 posts

Re: Cloudflare outage on June 21, 2022

#201

Post by mikewang »

I read the blog twice and have some thoughts: The root cause seems is as: "While deploying a change to our prefix advertisement policies, a re-ordering of terms caused us to withdraw a critical subset of prefixes."

And a dry-run: "a Change Request ticket was created, which includes a dry-run of the change, as well as a stepped rollout procedure."

And a Peer review: "Before it was allowed to go out, it was also peer reviewed by multiple engineers. "

I would doubt the expertise of tech guys of cloudflare, reviewing the change. And there was a dry-run.

But is it really OK to apply the change to a spine network which would affect 50% network traffic? Just out of peer review and a dry run? No green/blue, no gray release, maybe these are not proper for a small change here. But this "small" change really got big affect. I thougt it was worth it.

And from my shallow experience, the dry-run would always have do nothing to the env. It is dry-run anyway.

And at last the three lines are found out. So I wonder how did this re-order happen? And why?

With these tiny changes, there should be some mechanism to verify their correctness, not just review and dry-run.

Re: Cloudflare outage on June 21, 2022

#202

Post by jrockway »

jiggawatts wrote: post #180

The default way that most networking devices are managed is crazy in this day and age. Like the post-mortem says, they will put mitigations in place, but this is something every network admin has to implement bespoke after learning the hard way that the default management approach is dangerous. I’ve personally watched admins make routing changes where any error would cut them off from the device they are managing and…

> What should be the default on every networking device is a two-stage commit where the second stage requires a new TCP connection.

Always good. The system built into display settings ("click yes if you can read this, or the change will be reverted in 15 seconds") has saved me a number of times. No reason not to apply that to other settings where the data channel is the same as the control channel.

Re: Cloudflare outage on June 21, 2022

#203

Post by matsur »

mikewang wrote: post #201

I read the blog twice and have some thoughts: The root cause seems is as: "While deploying a change to our prefix advertisement policies, a re-ordering of terms caused us to withdraw a critical subset of prefixes." And a dry-run: "a Change Request ticket was created, which includes a dry-run of the change, as well as a stepped rollout procedure." And a Peer review: "Before it was allowed to go out, it was also peer r…

We use a phased rollout process for all routine changes (like this one). Once a change has passed peer review and the "dry-run", changes are rolled out to progressively larger slices of our production environment, with monitoring systems and engineers watching for adverse effects.

The specific network locations that were impacted by this change were amongst the last to see the change rolled out. One deficiency in our deployment strategy (which we will correct) is that no network locations in the affected "MCP" configuration received the change early in our rollout process. If that had been the case, we would have found the problem much earlier and the incident's impact would have been much reduced.

Re: Cloudflare outage on June 21, 2022

#204

Post by jhugo »

jiggawatts wrote: post #180

The default way that most networking devices are managed is crazy in this day and age. Like the post-mortem says, they will put mitigations in place, but this is something every network admin has to implement bespoke after learning the hard way that the default management approach is dangerous. I’ve personally watched admins make routing changes where any error would cut them off from the device they are managing and…

Even 20 years ago at a nationwide ISP we used to have a completely separate emergency management network over GSM. I'm amazed that at CF's scale management is still done the way described.

Re: Cloudflare outage on June 21, 2022

#205

Post by paranoidrobot »

daniel-grigg wrote: post #199
llama052 wrote: post #189

We use Cloudflare to serve ~20-30TB of traffic a month where I work. Was the SRE on call when I got paged on our blackbox monitoring/third party web checks failing.. It was very pleasant to find the cloudflare status page pointing me to the issue right away (minutes after our alerts triggered), even though I couldn't replicate the issue myself yet. I wish more companies would take note of the transparency and sense o…

Your experience is different to mine. Here in Australia the status page was inaccessible. Which immediately leads to common wisdom on monitoring systems - the monitor must always be separate to the monitored.

> Here in Australia the status page was inaccessible.

Me and a bunch of my colleagues were all able to access it (different ISPs) on the east coast.

Re: Cloudflare outage on June 21, 2022

#206

Post by kondro »

daniel-grigg wrote: post #199
llama052 wrote: post #189

We use Cloudflare to serve ~20-30TB of traffic a month where I work. Was the SRE on call when I got paged on our blackbox monitoring/third party web checks failing.. It was very pleasant to find the cloudflare status page pointing me to the issue right away (minutes after our alerts triggered), even though I couldn't replicate the issue myself yet. I wish more companies would take note of the transparency and sense o…

Your experience is different to mine. Here in Australia the status page was inaccessible. Which immediately leads to common wisdom on monitoring systems - the monitor must always be separate to the monitored.

Also in Australia and had no issues with the status page from the very beginning of the outage.

Re: Cloudflare outage on June 21, 2022

#207

Post by bad416f1f5a2 »

cowsandmilk wrote: post #77
kevin_nisbet wrote: post #41

Earlier quoted context omitted.

I agree, I think the transparency builds trust and I encourage it where I can. The counter thought I had when reading this case though, is it almost feels too fast. What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done. Doing incident response (both outage and security), the tactical fixes for a specific problem are usual…

I have to agree. The environment that leads to a fast blog post may also lead to this quote from the post: > This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. They are running as fast as they can and this extended the incident. There is a “slow is smooth, smooth is fast” lesson in here. I’d rather have a team that tak…

At a previous job, I worked with two guys who were excellent in a crisis. One of them used to run operations, the other was a crusty old programmer who’d been around for a while. I tried to learn as much as I could from both of them.

At around the same time, I was watching HBO’s The Wire. One scene had a high-profile shooting with a frantic police response; people were running everywhere trying to help. The lead commander on the scene gave this instruction to his sergeant: “Slow this thing down to a crawl. Give these bastards no chance to fuck up in a meaningful way.”

And then it hit me. That’s exactly how they ran the calls. I asked them about this, and they said absolutely – human nature, when things are broken badly, is biased towards action. You want to try to make a change, to reboot a system, to do something to hopefully make things better. But often if you were not careful, you risk losing information about the outage you are in. Best case scenario you luck into fixing the problem and don’t know how. Worst case? You’ve changed the state of an already broken system, and have done nothing but add more variables to unwind.

So now, every time I’m on an outage call, I try to do what Wes and Tim and Major Rawls would all do: I take control, pump the brakes, and make sure that we are capturing enough information about the current state that we don’t confuse ourselves further.

Re: Cloudflare outage on June 21, 2022

#208

Post by weird-eye-issue »

uhhhhuhyes wrote: post #128
herpderperator wrote: post #125

Earlier quoted context omitted.

I didn't quite understand this. It sounds like Cloudflare's outage didn't affect you depite being their customer. Why did their large outage not affect you?

Because of the time at which the outage occurred, most of this person's customers were not trying to access the site.

We still get several hundred pageviews per hour during the outage. It just didn't seem to effect us much for some reason (but the reason is not that nobody was going to our site anyways)

I did get an alert from Uptime Robot but when I checked everything was fine and so I thought it was a false positive.

Re: Cloudflare outage on June 21, 2022

#209

Post by kebab-case »

I lead the platform team of a fairly young startup in the D2C commerce space in the APAC region. This outage happened during peak traffic hours which made me and the team look like amateurs in the company.

Cloudflare is great, and I would never move away from it. But from a business continuity standpoint, is there a fallback approach that we should be prepared for during such cases?

One crude approach we were discussing is during an outage we could change the NS records in the registrar to point to for eg. Google Cloud DNS which would already be in sync in terms of the DNS records it has.

Re: Cloudflare outage on June 21, 2022

#210

Post by smugma »

saghm wrote: post #108
smugma wrote: post #60

Earlier quoted context omitted.

I agree that this is a free ad/recruitment. However, it’s easy to see how more conservative businesses see this as a risk. They are highlighting their deficiencies, letting their big important clients know that human error can bring their network down. Additionally, these post-mittens work for Cloudflare because they have a great reputation and good uptime. If this were happening daily or weekly, it would be a warnin…

OTOH, I think most actual engineers would know that everywhere has deficiencies and can be brought down by human error, and I'd personally rather use a product where the people running it admit this rather than just claim that their genius engineers made it 100% foolproof and nothing could ever possibly go wrong

100% agree. But…

1. On the buy side, crappy big companies with procurement etc. may not have “actual engineers” deciding things. Maybe for something like Cloudflare that’s likely to sit within an actual technical team’s mandate

2. Not 100% foolproof but if a startup is selling its tool and has an uptime page and details of all its downtime and as a prospective customer you go and see that they have 3-4 hour downtimes once or twice a month, it should raise alarm bells.

Post reply on HN