Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on June 21, 2022

blog.cloudflare.com

Post reply on HN
Print view View original

191–200 of 234 posts

Re: Cloudflare outage on June 21, 2022

#191

Post by dmix »

llama052 wrote: post #189

We use Cloudflare to serve ~20-30TB of traffic a month where I work. Was the SRE on call when I got paged on our blackbox monitoring/third party web checks failing.. It was very pleasant to find the cloudflare status page pointing me to the issue right away (minutes after our alerts triggered), even though I couldn't replicate the issue myself yet. I wish more companies would take note of the transparency and sense o…

> I wish more companies would take note of the transparency and sense of urgency on updating their status page.

Looking at you Twilio...

Re: Cloudflare outage on June 21, 2022

#192

Post by sarathyweb »

dmix wrote: post #191
llama052 wrote: post #189

We use Cloudflare to serve ~20-30TB of traffic a month where I work. Was the SRE on call when I got paged on our blackbox monitoring/third party web checks failing.. It was very pleasant to find the cloudflare status page pointing me to the issue right away (minutes after our alerts triggered), even though I couldn't replicate the issue myself yet. I wish more companies would take note of the transparency and sense o…

> I wish more companies would take note of the transparency and sense of urgency on updating their status page. Looking at you Twilio...

Looking at you, AWS

Re: Cloudflare outage on June 21, 2022

#193

Post by crvdgc »

yuliyp wrote: post #137
gouggoug wrote: post #126

Earlier quoted context omitted.

I'd be super interested in understanding what this means concretely. For example, are we talking about reverting commits? If so, why were engineers reverting reverts?

Developer 1 fetches code, changes flag A. Rebuilds config. Developer 2 fetches code, changes flag B. Rebuilds config. Developer 1 deploys built config. Developer 2 deploys built config, inadvertently reverts developer 1's changes.

Sounds like a racing condition. A lock (algorithmical or just through communication) should have been used.

Re: Cloudflare outage on June 21, 2022

#194

Post by nickjj »

meibo wrote: post #71
mproud wrote: post #63

If I use Cloudflare, what can I do — if anything — to avoid disruption when they go down?

On the enterprise plans, you are able to set up your own DNS server that can route users away from Cloudflare, either to your origin or to another CDN/proxy.

This comes with a major caveat.

Your DNS host needs to support being able to assign a CNAME record on your root domain to a domain provided by Cloudflare. AWS Route 53 does not let you do this which I imagine is a decent chunk of enterprise clients. AWS only lets you alias records to AWS resources not external domains.

With that said, even with enterprise in this case you would need to go all-in with Cloudlfare's nameservers or run the risk of not having DDoS protection on your root domain (ie. example.com wouldn't be protected but you could protect www.example.com since a CNAME with subdomains is a standard thing).

However it's kind of interesting because an attacker could get the real IP of your root domain's AWS load balancer which is probably the same load balancer used for the `www` version of your site too, but now that they know your load balancer's IP they can completely bypass Cloudflare and go straight to your infrastructure.

I'm pretty sure AWS doesn't let you assign an external domain with their aliases because they want you to pay them for AWS Shield Advanced instead of using Cloudflare because AWS knows getting an enterprise client to change their nameservers and all of their DNS records (potentially dozens of domains and multiple hundreds of records) is kind of a pain. It can be done but it's a friction point.

Re: Cloudflare outage on June 21, 2022

#196

Post by forrestthewoods »

techsupporter wrote: post #160
forrestthewoods wrote: post #144

Earlier quoted context omitted.

Some tools are more fragile and error prone than others.

Except that this wasn't an example of BGP being prone to error or fragile. This was, as the blog post specifically calls out, human error. They put two BGP announcement rules after the "deny everything not previously allowed" rule. It's the same as if someone did this to a set of ACLs on a firewall. The main difference between BGP and all other tools is that if you mess up BGP, you've done a very visible thing becaus…

Some tools are more prone to human error than others.

Another canonical example is C++. Some tools make it easy to blow your leg off. Some tools provide safety mechanisms to stop the saw from cutting off your finger.

Re: Cloudflare outage on June 21, 2022

#198

Post by exikyut »

jgrahamc wrote: post #33
thejosh wrote: post #5

07:42: The last of the reverts has been completed. This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. Ouch

Well, the "we can't reach these data centers at all and need to go through the break glass procedure" was pretty "ouch" also.

I can('t) imagine, yikes.

Now I'm remembering the story of how, when a certain blue website fell off the Internet for a day ~a decade and a half ago (due to some slightly broken database migration logic), out-of-band access boiled down to who was still logged in (!): https://rachelbythebay.com/w/2019/01/20/quiet/

Amusingly when things went wrong again last year, it was BGP's fault (is this the hyperscale equivalent of "it's always DNS" or something?). Engineers (with adequate credentials) had to actually drive to the datacenter haha.

I would be very interested to hear more about how the break-glass process worked.

Re: Cloudflare outage on June 21, 2022

#199

Post by daniel-grigg »

llama052 wrote: post #189

We use Cloudflare to serve ~20-30TB of traffic a month where I work. Was the SRE on call when I got paged on our blackbox monitoring/third party web checks failing.. It was very pleasant to find the cloudflare status page pointing me to the issue right away (minutes after our alerts triggered), even though I couldn't replicate the issue myself yet. I wish more companies would take note of the transparency and sense o…

Your experience is different to mine. Here in Australia the status page was inaccessible. Which immediately leads to common wisdom on monitoring systems - the monitor must always be separate to the monitored.

Re: Cloudflare outage on June 21, 2022

#200

Post by matsur »

systemvoltage wrote: post #162
grenbys wrote: post #161

Would be great if the timeline covered 19 minutes of 6:32 – 06:51. How long did it take to get the right people on the call? How long did it take to identify deployment as a suspect? Another massive gap is the rollback: 6:58 – 7:42 – 44 minutes! What exactly was going on and why did it take so long? What were those back-up procedures mentioned briefly? Why engineers where stepping on each other toes? What's the story…

It would be fun to be a fly on the wall when shit hits the fan in general. From Nuclear meltdowns to 9/11 ATC recordings, it is fascinating to see how emergencies play out and what kind of things go on with boots-on-ground, all-hands-on-deck situations. Like, does Cloudflare have an emergency procedure for escalation? What does that look like? How does the CTO get woken up in the middle of the night? How to get in to…

We spend a lot of time and thought building out our incident management processes and tooling. We were not making things up as we went last night.

https://sre.google/resources/book-update/managing-incidents/ is Google focused, but our flavor of incident response is not too far off.

Post reply on HN