Earlier quoted context omitted.
> since engineers got locked out of their systems Sounds like the same happened here: "Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. We have backup procedures for handling such an event and used them to take control of the affected locations." But Cloudflare had sufficient backup connectivity to fix it. I'm curious how Cl…
> the solution long ago was always a modem on an auxiliary port Now you can use mobile Internet (4G/5G)
Cloudflare outage on June 21, 2022
171–180 of 234 posts
Re: Cloudflare outage on June 21, 2022
#172One of our sites uses Cloudflare and serves 400k pageviews per month and generates around $650/day in ad and affiliate revenue. If the site is not up the business is not making any money. Looking at the hourly chart in Google Analytics (compared to the previous day) there isn't even a blip during this outage. So for all the advantages we get from Cloudflare (caching, WAF, security [our WP admin is secured with Cloudf…
Re: Cloudflare outage on June 21, 2022
#173Yet another BGP caused outage. At some point we should collect all of them: - Cloudflare 2022 (this one) - Facebook 2021: https://news.ycombinator.com/item?id=28752131 - this one probably had the single biggest impact, since engineers got locked out of their systems, which made the fixing part look like a sci-fi movie - (Indirectly caused by BGP: Cloudflare 2020: https://blog.cloudflare.com/cloudflare-outage-on-july-…
> since engineers got locked out of their systems Sounds like the same happened here: "Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. We have backup procedures for handling such an event and used them to take control of the affected locations." But Cloudflare had sufficient backup connectivity to fix it. I'm curious how Cl…
Re: Cloudflare outage on June 21, 2022
#174It sounds like it's a key architectural part of the system that "[...] convert all of our busiest locations to a more flexible and resilient architecture."
25 year experience and it's always the things that are supposed to make us "more flexible" and "more resilient" or robust/stable/safer that ends up royally f'ing us where the light don't shine.
Re: Cloudflare outage on June 21, 2022
#175BGP changes should be like the display resolution changes on your PC... It should revert as a failsafe if not confirmed within X minutes.
$ apply new rules; sleep 10; apply original rules
If your ssh access was still working and various sites were still up during that 10sec you were probably good to go - or at least you hadn't shut yourself out.
Re: Cloudflare outage on June 21, 2022
#176BGP changes should be like the display resolution changes on your PC... It should revert as a failsafe if not confirmed within X minutes.
I'm sure there were and are better ways of doing it, but it was simple enough and worked for us.
Re: Cloudflare outage on June 21, 2022
#177BGP changes should be like the display resolution changes on your PC... It should revert as a failsafe if not confirmed within X minutes.
Back when I was a briefly a network engineer at the start of my career, on cisco equipment we'd do 'reload in 5' before big changes - so it'd auto restart after 5 minutes unless cancelled. I'm sure there were and are better ways of doing it, but it was simple enough and worked for us.
junipers allows for instance, one to do the command commit confirmed, which will apply the configuration, and revert back to the previous version if one does not acknowledge this command within a predifined time. this prevents permanent lockout out of a system.
Re: Cloudflare outage on June 21, 2022
#178Earlier quoted context omitted.
>I feel like others lose opportunities by not doing the same IMO it is a slippery slope to see this as opportunity too strongly. Sure, doing the right thing may be net beneficial to the business in the long run...but the $RIGHT_THING should be done first and foremost because it's the right thing.
I believe Marcus Aurelius had something similar to say on the matter. :-)
Re: Cloudflare outage on June 21, 2022
#179Earlier quoted context omitted.
It wasn't a global outage.
I thought it was global? 19 data centers were taken offline which "handle a significant proportion of [Cloudflare's] global traffic".
Re: Cloudflare outage on June 21, 2022
#180Like the post-mortem says, they will put mitigations in place, but this is something every network admin has to implement bespoke after learning the hard way that the default management approach is dangerous.
I’ve personally watched admins make routing changes where any error would cut them off from the device they are managing and prevent them from rolling it back — pretty much what happened here.
What should be the default on every networking device is a two-stage commit where the second stage requires a new TCP connection.
Many devices still rely on “not saving” the configuration, with a power cycle as the rollback to the previous saved state. This is a great way to turn a small outage into a big one.
This style of device management may have been okay for small office routers where you can just walk into the “server closet” to flip the switch. It was okay in the era when device firmware was measured in kilobytes and boot times in single digit seconds.
Globally distributed backbone routers are an entirely different scenario but the manufacturers use the same outdated management concepts!
(I have seen some small improvements in this space, such as devices now keeping a history of config files by default instead of a single current-state file only.)