Live data from Hacker News

Cloudflare outage on June 21, 2022

blog.cloudflare.com

171–180 of 234 posts

Re: Cloudflare outage on June 21, 2022

#171
post #159
post #103

Earlier quoted context omitted.

> since engineers got locked out of their systems Sounds like the same happened here: "Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. We have backup procedures for handling such an event and used them to take control of the affected locations." But Cloudflare had sufficient backup connectivity to fix it. I'm curious how Cl…

> the solution long ago was always a modem on an auxiliary port Now you can use mobile Internet (4G/5G)

Cell coverage inside datacenters isn't always suitable, occasionally even by-design.

Re: Cloudflare outage on June 21, 2022

#172

One of our sites uses Cloudflare and serves 400k pageviews per month and generates around $650/day in ad and affiliate revenue. If the site is not up the business is not making any money. Looking at the hourly chart in Google Analytics (compared to the previous day) there isn't even a blip during this outage. So for all the advantages we get from Cloudflare (caching, WAF, security [our WP admin is secured with Cloudf…

Would you mind sharing which site that is?

Re: Cloudflare outage on June 21, 2022

#173
post #103

Yet another BGP caused outage. At some point we should collect all of them: - Cloudflare 2022 (this one) - Facebook 2021: https://news.ycombinator.com/item?id=28752131 - this one probably had the single biggest impact, since engineers got locked out of their systems, which made the fixing part look like a sci-fi movie - (Indirectly caused by BGP: Cloudflare 2020: https://blog.cloudflare.com/cloudflare-outage-on-july-…

> since engineers got locked out of their systems Sounds like the same happened here: "Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. We have backup procedures for handling such an event and used them to take control of the affected locations." But Cloudflare had sufficient backup connectivity to fix it. I'm curious how Cl…

They have their machines also connected to another AS, so when their network doesn't/can't route, they can still get to their machines to fix stuff.

Re: Cloudflare outage on June 21, 2022

#174
How did no one at cloudflare think that this MCP thing should be part of the staging rollout? I imagine that was part of a // TODO.

It sounds like it's a key architectural part of the system that "[...] convert all of our busiest locations to a more flexible and resilient architecture."

25 year experience and it's always the things that are supposed to make us "more flexible" and "more resilient" or robust/stable/safer that ends up royally f'ing us where the light don't shine.

Re: Cloudflare outage on June 21, 2022

#175

BGP changes should be like the display resolution changes on your PC... It should revert as a failsafe if not confirmed within X minutes.

There was a common pattern in use back in the day when I managed openbsd filewalls (can't remember if it was ipf or pf days). When changing firewall rules over ssh, you'd use a command line like:

$ apply new rules; sleep 10; apply original rules

If your ssh access was still working and various sites were still up during that 10sec you were probably good to go - or at least you hadn't shut yourself out.

Re: Cloudflare outage on June 21, 2022

#176

BGP changes should be like the display resolution changes on your PC... It should revert as a failsafe if not confirmed within X minutes.

Back when I was a briefly a network engineer at the start of my career, on cisco equipment we'd do 'reload in 5' before big changes - so it'd auto restart after 5 minutes unless cancelled.

I'm sure there were and are better ways of doing it, but it was simple enough and worked for us.

Re: Cloudflare outage on June 21, 2022

#177
post #176

BGP changes should be like the display resolution changes on your PC... It should revert as a failsafe if not confirmed within X minutes.

Back when I was a briefly a network engineer at the start of my career, on cisco equipment we'd do 'reload in 5' before big changes - so it'd auto restart after 5 minutes unless cancelled. I'm sure there were and are better ways of doing it, but it was simple enough and worked for us.

most ISP tier routers have an entire commit engine to load and apply configs.

junipers allows for instance, one to do the command commit confirmed, which will apply the configuration, and revert back to the previous version if one does not acknowledge this command within a predifined time. this prevents permanent lockout out of a system.

Re: Cloudflare outage on June 21, 2022

#178
post #167
post #145

Earlier quoted context omitted.

>I feel like others lose opportunities by not doing the same IMO it is a slippery slope to see this as opportunity too strongly. Sure, doing the right thing may be net beneficial to the business in the long run...but the $RIGHT_THING should be done first and foremost because it's the right thing.

I believe Marcus Aurelius had something similar to say on the matter. :-)

quodcumque erat ?

Re: Cloudflare outage on June 21, 2022

#179

Earlier quoted context omitted.

It wasn't a global outage.

I thought it was global? 19 data centers were taken offline which "handle a significant proportion of [Cloudflare's] global traffic".

I did not notice Cloudflare going down. Only reason I knew was because of this thread. Either it was because I was asleep, or my local PoP wasn't affected.

Re: Cloudflare outage on June 21, 2022

#180
The default way that most networking devices are managed is crazy in this day and age.

Like the post-mortem says, they will put mitigations in place, but this is something every network admin has to implement bespoke after learning the hard way that the default management approach is dangerous.

I’ve personally watched admins make routing changes where any error would cut them off from the device they are managing and prevent them from rolling it back — pretty much what happened here.

What should be the default on every networking device is a two-stage commit where the second stage requires a new TCP connection.

Many devices still rely on “not saving” the configuration, with a power cycle as the rollback to the previous saved state. This is a great way to turn a small outage into a big one.

This style of device management may have been okay for small office routers where you can just walk into the “server closet” to flip the switch. It was okay in the era when device firmware was measured in kilobytes and boot times in single digit seconds.

Globally distributed backbone routers are an entirely different scenario but the manufacturers use the same outdated management concepts!

(I have seen some small improvements in this space, such as devices now keeping a history of config files by default instead of a single current-state file only.)

Post reply on HN