Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on June 21, 2022

blog.cloudflare.com

Post reply on HN
Print view View original

211–220 of 234 posts

Re: Cloudflare outage on June 21, 2022

#211

Post by badcoder72684 »

jiggawatts wrote: post #180

The default way that most networking devices are managed is crazy in this day and age. Like the post-mortem says, they will put mitigations in place, but this is something every network admin has to implement bespoke after learning the hard way that the default management approach is dangerous. I’ve personally watched admins make routing changes where any error would cut them off from the device they are managing and…

> What should be the default on every networking device is a two-stage commit where the second stage requires a new TCP connection.

OpenWRT router distribution has had this for years, it's amazing! (As is OpenWRT)

OpenWRT also has SQM CAKE which saved my sanity on parents DSL connection for years. As far as congestion control and bandwidth sharing goes, nothing else compares

Re: Cloudflare outage on June 21, 2022

#212

Post by cyphar »

daniel-grigg wrote: post #199
llama052 wrote: post #189

We use Cloudflare to serve ~20-30TB of traffic a month where I work. Was the SRE on call when I got paged on our blackbox monitoring/third party web checks failing.. It was very pleasant to find the cloudflare status page pointing me to the issue right away (minutes after our alerts triggered), even though I couldn't replicate the issue myself yet. I wish more companies would take note of the transparency and sense o…

Your experience is different to mine. Here in Australia the status page was inaccessible. Which immediately leads to common wisdom on monitoring systems - the monitor must always be separate to the monitored.

Are you using Cloudflare DNS? I couldn't access it either during the outage and I just came to the realisation it might be because I switched to 1.1.1.1 a while ago.

Re: Cloudflare outage on June 21, 2022

#213

Post by Maxious »

paranoidrobot wrote: post #205
daniel-grigg wrote: post #199

Earlier quoted context omitted.

Your experience is different to mine. Here in Australia the status page was inaccessible. Which immediately leads to common wisdom on monitoring systems - the monitor must always be separate to the monitored.

> Here in Australia the status page was inaccessible. Me and a bunch of my colleagues were all able to access it (different ISPs) on the east coast.

It seemed like ISPs with east coast Vocus backhaul (going to SYD rather than MEL/PER POPs - TIL there's also CBR/BNE/ADL) had it worst trying to get to the status page but as the timeline shows, cloudflare was probably already recovering as we were triaging.

Re: Cloudflare outage on June 21, 2022

#214

Post by prophesi »

jgrahamc wrote: post #168
prophesi wrote: post #163

Earlier quoted context omitted.

I thought it was global? 19 data centers were taken offline which "handle a significant proportion of [Cloudflare's] global traffic".

I am in Lisbon and was not having trouble because Cloudflare's Lisbon data center was not affected. But over in Madrid there was trouble. It depended where you are.

Gotcha, so it was out internationally, but selectively.

Re: Cloudflare outage on June 21, 2022

#215

Post by beefsack »

paranoidrobot wrote: post #205
daniel-grigg wrote: post #199

Earlier quoted context omitted.

Your experience is different to mine. Here in Australia the status page was inaccessible. Which immediately leads to common wisdom on monitoring systems - the monitor must always be separate to the monitored.

> Here in Australia the status page was inaccessible. Me and a bunch of my colleagues were all able to access it (different ISPs) on the east coast.

I was also able to access it, and my primary DNS was 1.1.1.1, though I use 8.8.8.8 as secondary so that might've saved me if primary wasn't working.

Re: Cloudflare outage on June 21, 2022

#216

Post by icehawk »

jiggawatts wrote: post #180

The default way that most networking devices are managed is crazy in this day and age. Like the post-mortem says, they will put mitigations in place, but this is something every network admin has to implement bespoke after learning the hard way that the default management approach is dangerous. I’ve personally watched admins make routing changes where any error would cut them off from the device they are managing and…

If what's presented in the blog is actual configuration that's very much a junos device and definitely has the ability to have commit with confirm, auto rollback, and commit history. Not using it is more of an issue with the automation, as they said.

(Which is an issue with the /automation/ defaults. I've learned enough to do commit confirm, but by default ansible does a hard commit.)

Re: Cloudflare outage on June 21, 2022

#217

Post by juancb »

kebab-case wrote: post #209

I lead the platform team of a fairly young startup in the D2C commerce space in the APAC region. This outage happened during peak traffic hours which made me and the team look like amateurs in the company. Cloudflare is great, and I would never move away from it. But from a business continuity standpoint, is there a fallback approach that we should be prepared for during such cases? One crude approach we were discuss…

Cedexis now part of Citrix offers a multi-cdn product, that allows you to load balance between CDNs, check them out. I'm not sure who else is in that space, but in general it sounds like you want a multi-cdn strategy.

Re: Cloudflare outage on June 21, 2022

#218

Post by [deleted] »

beefsack wrote: post #215
paranoidrobot wrote: post #205

Earlier quoted context omitted.

> Here in Australia the status page was inaccessible. Me and a bunch of my colleagues were all able to access it (different ISPs) on the east coast.

I was also able to access it, and my primary DNS was 1.1.1.1, though I use 8.8.8.8 as secondary so that might've saved me if primary wasn't working.

[deleted]

Re: Cloudflare outage on June 21, 2022

#219

Post by trhway »

crvdgc wrote: post #193
yuliyp wrote: post #137

Earlier quoted context omitted.

Developer 1 fetches code, changes flag A. Rebuilds config. Developer 2 fetches code, changes flag B. Rebuilds config. Developer 1 deploys built config. Developer 2 deploys built config, inadvertently reverts developer 1's changes.

Sounds like a racing condition. A lock (algorithmical or just through communication) should have been used.

Lock is another huge failure mode.

Re: Cloudflare outage on June 21, 2022

#220

Post by bandwidth »

kebab-case wrote: post #209

I lead the platform team of a fairly young startup in the D2C commerce space in the APAC region. This outage happened during peak traffic hours which made me and the team look like amateurs in the company. Cloudflare is great, and I would never move away from it. But from a business continuity standpoint, is there a fallback approach that we should be prepared for during such cases? One crude approach we were discuss…

If you're okay with load balancing DNS queries across multiple providers you could do 2x Cloudflare primary NS, 2x GCP for example where each provider is in sync with each other.

If not, a manual swap at the registrar level would be good enough.

I should also mention this approach sort of breaks with Cloudflare's proxied records which dynamically assign anycast IPs for records placed on their CDN. So if using this approach the failover NS provider would probably need to also use a different CDN, preferably one that just gives you a CNAME.

Post reply on HN