Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage – 24 hours now

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 76 posts

Re: Cloudflare outage – 24 hours now

#21

Post by adrr »

sailingparrot wrote: post #16

I never experienced a longer than 12 hours outage with any service provider over my ~13 years career (maybe I was lucky). But thanks to Cloudflare I have been able to enjoy not just one, but two ~24h outages in not even a month! Jokes aside, it must be extremely stressful to be a SRE at CF recently. But something is clearly wrong over there. We have been burned so bad there is no chance we will touch CF ever again in…

Azure leap year outage is a famous one.

https://azure.microsoft.com/en-us/blog/summary-of-windows-az...

Re: Cloudflare outage – 24 hours now

#22

Post by 0x0000000 »

issafram wrote: post #8
spacebacon wrote: post #5

Hmm... Who just changed their dns vs. riding it out?

It hasn't affected my home network at all. I use their DNS servers and nothing has resolving addresses has not stopped working

Parent comment was likely referring to authoritative DNS, not Cloudflare's public resolvers.

Re: Cloudflare outage – 24 hours now

#23

Post by sailingparrot »

CodesInChaos wrote: post #20
sailingparrot wrote: post #16

I never experienced a longer than 12 hours outage with any service provider over my ~13 years career (maybe I was lucky). But thanks to Cloudflare I have been able to enjoy not just one, but two ~24h outages in not even a month! Jokes aside, it must be extremely stressful to be a SRE at CF recently. But something is clearly wrong over there. We have been burned so bad there is no chance we will touch CF ever again in…

> two ~24h outages in not even a month Wasn't the previous outage on Oct 30 less than an hour?

Yep, but on Oct 9 they were down for 22h.

Re: Cloudflare outage – 24 hours now

#24

Post by ta1243 »

sailingparrot wrote: post #23
CodesInChaos wrote: post #20

Earlier quoted context omitted.

> two ~24h outages in not even a month Wasn't the previous outage on Oct 30 less than an hour?

Yep, but on Oct 9 they were down for 22h.

Trying high for that 2-nines reliability.

You just can't get that level of reliability if you do it yourself, no matter how hard you try.

Re: Cloudflare outage – 24 hours now

#25

Post by ta1243 »

nkcmr wrote: post #13
tootie wrote: post #6

I've heard tell of massive DDoS attacks against international news sources (AP, Reuters, NY Times). Not sure if this is related.

In this case it is not. A power outage in a critical data-center is the root cause here: https://www.cloudflarestatus.com/incidents/hm7491k53ppg

If you can't cope with the loss of a data centre you're not really running a resilient system.

Re: Cloudflare outage – 24 hours now

#26

Post by sph »

Looking forward to a more decentralised global Internet, with packets being routed through alternative paths, so outages like these become a non-event.

I understand we do not have the technology for that just yet, and DevOps able to configure TLS terminators on their own are worth their weight in gold.

Hard to imagine how the Internet could ever exist without Cloudflare.

Re: Cloudflare outage – 24 hours now

#28

Post by tux3 »

nkcmr wrote: post #13
tootie wrote: post #6

I've heard tell of massive DDoS attacks against international news sources (AP, Reuters, NY Times). Not sure if this is related.

In this case it is not. A power outage in a critical data-center is the root cause here: https://www.cloudflarestatus.com/incidents/hm7491k53ppg

What I'm confused by is we had "power partially restored" 22 hours ago, and no news from PDX02 since.

I assume both Clouflare and Flexential are on DEFCON 1 right now, but I'm wondering if it might be more than just the building going dark.

There's something about a failover than was attempted and crashed halfway through, but unclear if that's what's causing the 24h+ situation.

Re: Cloudflare outage – 24 hours now

#30

Post by summarity »

sanat wrote: post #12

I run hirevire.com one way video interview SaaS - and we were pretty much dead in the water during the Cloudflare Stream outage. We moved out to BunnyCDN's stream after waiting for 20 hours. One side benefit is that our videos are now stored in EU instead of Cloudflare's edge location around you.

I've also been using Bunny's image and video delivery, while using CF for everything else. It's pretty neat - it just works. I like having both in my toolbelt, makes fallbacks like these easy.
Post reply on HN