Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage – 24 hours now

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 76 posts

Re: Cloudflare outage – 24 hours now

#31

Post by sailingparrot »

ta1243 wrote: post #24
sailingparrot wrote: post #23

Earlier quoted context omitted.

Yep, but on Oct 9 they were down for 22h.

Trying high for that 2-nines reliability. You just can't get that level of reliability if you do it yourself, no matter how hard you try.

We won't do it ourselves, but we also won't do it with a provider that has accumulated 50+ hours of downtime in less than a month all the while having no communication or support.

That's barely clearing the one nine availability for the last 30 days (93%) for our particular stack on CF, this is insane.

Mind you last time we were hit by a 22h outage on Oct. 9 we didn't get so much as an email from CF either during or after the outage.

Re: Cloudflare outage – 24 hours now

#33

Post by itslennysfault »

ta1243 wrote: post #25
nkcmr wrote: post #13

Earlier quoted context omitted.

In this case it is not. A power outage in a critical data-center is the root cause here: https://www.cloudflarestatus.com/incidents/hm7491k53ppg

If you can't cope with the loss of a data centre you're not really running a resilient system.

On yesterday's post someone that used to work at CF mentioned that PDX is "the brain" and if it goes down data stops propagating and starts getting stale. It's crazy to me that a company that is so critical to SO MUCH of the traffic on the internet doesn't even have a failover strategy for "the brain" of their operation.

Re: Cloudflare outage – 24 hours now

#34

Post by kijin »

hipadev23 wrote: post #29

Is there a summary of what Cloudflare services are operational? Feels like it would be easier to track.

Basic proxying seems to be working fine for me. Existing DNS records continue to be served. Existing files on R2 are accessible. Can't change anything without a bunch of API errors, though. Hope I don't need to turn on "I'm under attack" anytime soon.

Re: Cloudflare outage – 24 hours now

#35

Post by throaway920181 »

sailingparrot wrote: post #16

I never experienced a longer than 12 hours outage with any service provider over my ~13 years career (maybe I was lucky). But thanks to Cloudflare I have been able to enjoy not just one, but two ~24h outages in not even a month! Jokes aside, it must be extremely stressful to be a SRE at CF recently. But something is clearly wrong over there. We have been burned so bad there is no chance we will touch CF ever again in…

> But something is clearly wrong over there

We renewed our agreement with them in the middle of the year (~$50k) and they've yet to invoice us for it. Our financial controller noticed and I pinged our account rep a few times. Not a peep back.

Re: Cloudflare outage – 24 hours now

#36

Post by renonce »

sph wrote: post #26

Looking forward to a more decentralised global Internet, with packets being routed through alternative paths, so outages like these become a non-event. I understand we do not have the technology for that just yet, and DevOps able to configure TLS terminators on their own are worth their weight in gold. Hard to imagine how the Internet could ever exist without Cloudflare.

The Internet has been decentralized from the beginning. Now I don't want to claim that Cloudflare made something worse (at least it's enabling a lot of websites to exist without fear of DDoS) but the fact is that Cloudflare made it more centralized, as there are lots of websites that cannot be accessed without going through Cloudflare.

Re: Cloudflare outage – 24 hours now

#37

Post by supertron »

samlinnfer wrote: post #19
perryizgr8 wrote: post #17

Earlier quoted context omitted.

Data point of just one, but my tunnels are working just fine.

If you've previously created a tunnel it will still work, just don't close it because you won't be able to open a new one.

Yeah, can confirm the same here. This seems to be the most frequent failure mode that I see with my use of Tunnels. Existing connections will continue working, but attempts to create new ones (or other changes) fail. Ie. the control plane fails more often than the core connectivity service.

I certainly prefer that failure mode to the opposite, but I do find the status information on Cloudflare's page to be very confusing about this.

Tunnels as a product is essentially heavily degraded (putting it lightly) and yet it's listed currently as "Cloudflare Tunnel: restored" [0]

https://www.cloudflarestatus.com [0]

Edit: also having used Shopify's CLI a little, one thing I noticed immediately is how opaque the whole thing is. They want to push you down a very specific path, and don't provide a lot of information if you want to take a bit more control over your dev process (as I always want to do) which directly leads to points of failure like this. From your GitHub links it looks like devs are struggling to figure out how to quickly switch to a different reverse proxy.

Re: Cloudflare outage – 24 hours now

#38

Post by samtho »

sailingparrot wrote: post #31
ta1243 wrote: post #24

Earlier quoted context omitted.

Trying high for that 2-nines reliability. You just can't get that level of reliability if you do it yourself, no matter how hard you try.

We won't do it ourselves, but we also won't do it with a provider that has accumulated 50+ hours of downtime in less than a month all the while having no communication or support. That's barely clearing the one nine availability for the last 30 days (93%) for our particular stack on CF, this is insane. Mind you last time we were hit by a 22h outage on Oct. 9 we didn't get so much as an email from CF either during or…

The no communication or support is the real killer here, imo. I can understand them having some catastrophic issues which I would be reasonably confident they could fix, it’s the uncertainty of the situation that makes me worry. Is this so easy to fix it will be back in less than an hour and they will communicate then? Are they going dark and I need to find a new provider asap?

Re: Cloudflare outage – 24 hours now

#40

Post by TeMPOraL »

sph wrote: post #26

Looking forward to a more decentralised global Internet, with packets being routed through alternative paths, so outages like these become a non-event. I understand we do not have the technology for that just yet, and DevOps able to configure TLS terminators on their own are worth their weight in gold. Hard to imagine how the Internet could ever exist without Cloudflare.

> decentralised global Internet, with packets being routed through alternative paths

> I understand we do not have the technology for that just yet

I looked at my router, remembered the term "packet-switched network", and wept.

Post reply on HN