Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage – 24 hours now

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 76 posts

Re: Cloudflare outage – 24 hours now

#51

Post by olivermuty »

sailingparrot wrote: post #31
ta1243 wrote: post #24

Earlier quoted context omitted.

Trying high for that 2-nines reliability. You just can't get that level of reliability if you do it yourself, no matter how hard you try.

We won't do it ourselves, but we also won't do it with a provider that has accumulated 50+ hours of downtime in less than a month all the while having no communication or support. That's barely clearing the one nine availability for the last 30 days (93%) for our particular stack on CF, this is insane. Mind you last time we were hit by a 22h outage on Oct. 9 we didn't get so much as an email from CF either during or…

To be fair, their status page says emails don’t work haha

Re: Cloudflare outage – 24 hours now

#52

Post by throwaway_13140 »

tux3 wrote: post #28
nkcmr wrote: post #13

Earlier quoted context omitted.

In this case it is not. A power outage in a critical data-center is the root cause here: https://www.cloudflarestatus.com/incidents/hm7491k53ppg

What I'm confused by is we had "power partially restored" 22 hours ago, and no news from PDX02 since. I assume both Clouflare and Flexential are on DEFCON 1 right now, but I'm wondering if it might be more than just the building going dark. There's something about a failover than was attempted and crashed halfway through, but unclear if that's what's causing the 24h+ situation.

if it was a state sponsored attack, various 3 letter agencies may want to investigate which could slow down restoration of service

Re: Cloudflare outage – 24 hours now

#54

Post by dogweather »

creshal wrote: post #3

I'm really looking forward to the post-mortem to this.

I can't believe we haven't heard anything yet. AFAIK we've only been told, "power outage", which was resolved yesterday.

What should our expectations be? The best assumption could be that this is the new normal.

Re: Cloudflare outage – 24 hours now

#55

Post by dogweather »

stpe wrote: post #50
l5870uoo9y wrote: post #32

Wonder if this is related to the many product launches recently? Even though my general impression is that they test rigorously with long-running alpha and beta test phases.

It is the consequences of a power outage in Flexential PDX02 data center.

That's the inference, but AFAIK there's been no direct assertion or explanation: Why has CF been knocked back to Alpha-status reliability across the board.

Re: Cloudflare outage – 24 hours now

#56

Post by dogweather »

spacebacon wrote: post #5

Hmm... Who just changed their dns vs. riding it out?

I'm planning my transition away for 10 or so subdomains and 30 records.

The only feature I need to research in new providers is: access to Whois ASN numbers, which I insert into HTTP request headers. I use this to tailor my site for .gov and .edu users.

Re: Cloudflare outage – 24 hours now

#57

Post by dogweather »

ta1243 wrote: post #25
nkcmr wrote: post #13

Earlier quoted context omitted.

In this case it is not. A power outage in a critical data-center is the root cause here: https://www.cloudflarestatus.com/incidents/hm7491k53ppg

If you can't cope with the loss of a data centre you're not really running a resilient system.

Especially 15 hours after the DC comes back up and you're still offline.

Re: Cloudflare outage – 24 hours now

#58

Post by smugma »

sailingparrot wrote: post #31
ta1243 wrote: post #24

Earlier quoted context omitted.

Trying high for that 2-nines reliability. You just can't get that level of reliability if you do it yourself, no matter how hard you try.

We won't do it ourselves, but we also won't do it with a provider that has accumulated 50+ hours of downtime in less than a month all the while having no communication or support. That's barely clearing the one nine availability for the last 30 days (93%) for our particular stack on CF, this is insane. Mind you last time we were hit by a 22h outage on Oct. 9 we didn't get so much as an email from CF either during or…

we used to joke that we had 5 8's of availability

Re: Cloudflare outage – 24 hours now

#59

Post by lamroger »

samlinnfer wrote: post #10

BTW Cloudflare tunnels are not working (for the at least the last 16 hours), but it says "Operational" and "restored" on the ticket. Since Shopify's CLI uses Cloudflare tunnels by default to load local resources, Shopify partners are affected by this outage by unable to develop apps, unless they use another tunnel: [0] https://github.com/Shopify/cli/issues/3065 [1] https://github.com/Shopify/cli/issues/3060

Wanted to hack today but the universe is telling me to go enjoy the sun

Re: Cloudflare outage – 24 hours now

#60

Post by barbazoo »

sph wrote: post #26

Looking forward to a more decentralised global Internet, with packets being routed through alternative paths, so outages like these become a non-event. I understand we do not have the technology for that just yet, and DevOps able to configure TLS terminators on their own are worth their weight in gold. Hard to imagine how the Internet could ever exist without Cloudflare.

> Looking forward to a more decentralised global Internet, with packets being routed through alternative paths, so outages like these become a non-event.

It's not just packet routing though, many of their other products seem to be affected as well.

Post reply on HN