Cloudflare outage on June 21, 2022 • Hacker News Board

Re: Cloudflare outage on June 21, 2022

#181

Post by nijave » Tue, Jun 21, 2022, 10:36 PM UTC

mlyle wrote: post #103

kurtextrem wrote: post #40

Yet another BGP caused outage. At some point we should collect all of them: - Cloudflare 2022 (this one) - Facebook 2021: https://news.ycombinator.com/item?id=28752131 - this one probably had the single biggest impact, since engineers got locked out of their systems, which made the fixing part look like a sci-fi movie - (Indirectly caused by BGP: Cloudflare 2020: https://blog.cloudflare.com/cloudflare-outage-on-july-…

> since engineers got locked out of their systems Sounds like the same happened here: "Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. We have backup procedures for handling such an event and used them to take control of the affected locations." But Cloudflare had sufficient backup connectivity to fix it. I'm curious how Cl…

Not sure how common it is, but you can get serial OOBM devices accessible over cellular which would then give you access to your equipment.

I'm surprised more places don't implement a "click here to confirm changes or it'll be rolled back in 5 minutes" like all those monitor settings dialogues

Re: Cloudflare outage on June 21, 2022

#182

Post by nijave » Tue, Jun 21, 2022, 10:40 PM UTC

solardev wrote: post #156

samwillis wrote: post #2

In a world where it can take weeks for other companies to publish a postmortem after an outage (if they ever do), I never ceases to amaze me how quickly CF manage to get something like this out. I think it's a testament to their Ops/Incident response teams and internal processes, it builds confidence in their ability to respond quickly when something does go wrong. Incredible work!

No provider is perfect, but it's because of stuff like this that I trust Cloudflare waaaaaaaaaaay more than the likes of Amazon. Transparency engenders trust, and eventually, love! Thank you, Cloudflare. The sheer level of technical competence of your engineering team continues to astound me. (Yes, they made a mistake and didn't catch an error in the diff. But your response process went exactly as it should, and your…

AWS is pretty decent if you're in an NDA contract (you have paid support). You can request RCAs for any incident you were impacted and they'll usually get them within a day.

Not as transparent as "post it on the internet" but at least better than the usual hand wavey bullshit

Re: Cloudflare outage on June 21, 2022

#183

Post by nijave » Tue, Jun 21, 2022, 10:48 PM UTC

grenbys wrote: post #161

Would be great if the timeline covered 19 minutes of 6:32 – 06:51. How long did it take to get the right people on the call? How long did it take to identify deployment as a suspect? Another massive gap is the rollback: 6:58 – 7:42 – 44 minutes! What exactly was going on and why did it take so long? What were those back-up procedures mentioned briefly? Why engineers where stepping on each other toes? What's the story…

Sounds like they had engineers connecting to the devices and manually rolling back changes. Something like...

Slack: "@here need to connect to to rollback change asap"

Re: Cloudflare outage on June 21, 2022

#184

Post by nijave » Tue, Jun 21, 2022, 10:51 PM UTC

alanning wrote: post #42

rubatuga wrote: post #38

Uh, shouldn’t there be a staging environment for these sort of changes?

Yes, that was one of the issues they mentioned in the post. Not that they didn’t have a staging/testing environment but that it didn’t include the specific type of new architecture configuration, “MCP”, that ultimately failed. One of their future changes is to include MCPs in their testing environments.

Ahh the old "dev doesn't quite match prod" issue

Re: Cloudflare outage on June 21, 2022

#185

Post by inferiorhuman » Tue, Jun 21, 2022, 10:53 PM UTC

jiggawatts wrote: post #180

The default way that most networking devices are managed is crazy in this day and age. Like the post-mortem says, they will put mitigations in place, but this is something every network admin has to implement bespoke after learning the hard way that the default management approach is dangerous. I’ve personally watched admins make routing changes where any error would cut them off from the device they are managing and…

The power cycle as a rollback is IMO reasonable. If you're talking about equipment in a data center you should presumably have some sort of remote power management on a separate network.

Alternatively some sort of watchdog timer would be a great addition (e.g. rollback within X minutes if the changes are not confirmed).

Re: Cloudflare outage on June 21, 2022

#186

Post by llama052 » Tue, Jun 21, 2022, 11:00 PM UTC

jiggawatts wrote: post #180

The default way that most networking devices are managed is crazy in this day and age. Like the post-mortem says, they will put mitigations in place, but this is something every network admin has to implement bespoke after learning the hard way that the default management approach is dangerous. I’ve personally watched admins make routing changes where any error would cut them off from the device they are managing and…

Luckily a lot of modern networking equipment has an automated rollback feature you can key off of. For instance junos based devices have a commit confirmed where it will auto rollback if not confirmed after X amount of time. Still pretty dated but they are designed to be reliable first.

Re: Cloudflare outage on June 21, 2022

#187

Post by jiggawatts » Tue, Jun 21, 2022, 11:01 PM UTC

inferiorhuman wrote: post #185

jiggawatts wrote: post #180

The default way that most networking devices are managed is crazy in this day and age. Like the post-mortem says, they will put mitigations in place, but this is something every network admin has to implement bespoke after learning the hard way that the default management approach is dangerous. I’ve personally watched admins make routing changes where any error would cut them off from the device they are managing and…

The power cycle as a rollback is IMO reasonable. If you're talking about equipment in a data center you should presumably have some sort of remote power management on a separate network. Alternatively some sort of watchdog timer would be a great addition (e.g. rollback within X minutes if the changes are not confirmed).

Again, you are talking about safety systems and mitigations that device customers “should” implement bespoke themselves.

I’m saying this is the problem — the device configuration approach should be safe by default.

Re: Cloudflare outage on June 21, 2022

#188

Post by drfrank » Tue, Jun 21, 2022, 11:01 PM UTC

Naively, it seems to me that there should at least be a warning somewhere if there are declarations after a REJECT-THE-REST.

I'm not familiar with whatever language this is, but wouldn't such a construct always indicate something was being ignored?

Re: Cloudflare outage on June 21, 2022

#189

Post by llama052 » Tue, Jun 21, 2022, 11:06 PM UTC

We use Cloudflare to serve ~20-30TB of traffic a month where I work. Was the SRE on call when I got paged on our blackbox monitoring/third party web checks failing..

It was very pleasant to find the cloudflare status page pointing me to the issue right away (minutes after our alerts triggered), even though I couldn't replicate the issue myself yet.

I wish more companies would take note of the transparency and sense of urgency on updating their status page. (Looking at you Azure)

Re: Cloudflare outage on June 21, 2022

#190

Post by inferiorhuman » Tue, Jun 21, 2022, 11:19 PM UTC

jiggawatts wrote: post #187

inferiorhuman wrote: post #185

Earlier quoted context omitted.

The power cycle as a rollback is IMO reasonable. If you're talking about equipment in a data center you should presumably have some sort of remote power management on a separate network. Alternatively some sort of watchdog timer would be a great addition (e.g. rollback within X minutes if the changes are not confirmed).

Again, you are talking about safety systems and mitigations that device customers “should” implement bespoke themselves. I’m saying this is the problem — the device configuration approach should be safe by default.

Yes/no – something like a watchdog timer should be part of whatever OS the router is running. Remote power management would be closer to bespoke but ought not to be too exotic.