Cloudflare outage – 24 hours now • Hacker News Board

Re: Cloudflare outage – 24 hours now

#61

Post by seized » Fri, Nov 03, 2023, 6:39 PM UTC

sangnoir wrote: post #48

itslennysfault wrote: post #33

Earlier quoted context omitted.

On yesterday's post someone that used to work at CF mentioned that PDX is "the brain" and if it goes down data stops propagating and starts getting stale. It's crazy to me that a company that is so critical to SO MUCH of the traffic on the internet doesn't even have a failover strategy for "the brain" of their operation.

Not great to know an aspiring hyperscaler is one tornado away from insolvency.

I dont think PDX would get many tornados BUT the PNW has the joys of earthquakes that are overdue.

Re: Cloudflare outage – 24 hours now

#62

Post by lazydon » Fri, Nov 03, 2023, 6:40 PM UTC

sph wrote: post #26

Looking forward to a more decentralised global Internet, with packets being routed through alternative paths, so outages like these become a non-event. I understand we do not have the technology for that just yet, and DevOps able to configure TLS terminators on their own are worth their weight in gold. Hard to imagine how the Internet could ever exist without Cloudflare.

Missing the /s I hope.

Re: Cloudflare outage – 24 hours now

#63

Post by client4 » Fri, Nov 03, 2023, 7:09 PM UTC

That are having issues with the new process spanning global MITM'd traffic to the NSA.

Re: Cloudflare outage – 24 hours now

#64

Post by [deleted] » Fri, Nov 03, 2023, 8:41 PM UTC

sph wrote: post #26

Looking forward to a more decentralised global Internet, with packets being routed through alternative paths, so outages like these become a non-event. I understand we do not have the technology for that just yet, and DevOps able to configure TLS terminators on their own are worth their weight in gold. Hard to imagine how the Internet could ever exist without Cloudflare.

[deleted]

Re: Cloudflare outage – 24 hours now

#65

Post by gtirloni » Fri, Nov 03, 2023, 8:44 PM UTC

dogweather wrote: post #53

Has Cloudflare said anything of substance yet? This is far beyond a simple power outage.

https://www.theregister.com/2023/11/02/cloudflare_outage/

""" In a nutshell, Cloudflare rolled out a new KV build to production. It turned out that the deployment tool had a bug, and some traffic got diverted to the wrong destination, which triggered a rollback … which failed. The result was that engineers had to manually switch the production route to the previous working version of Workers KV.

The problem is that an awful lot of Cloudflare products and services depend on Workers KV, meaning that when there is a problem with the platform, the blast radius can be impressive. """

Re: Cloudflare outage – 24 hours now

#66

Post by tux3 » Fri, Nov 03, 2023, 9:11 PM UTC

gtirloni wrote: post #65

dogweather wrote: post #53

Has Cloudflare said anything of substance yet? This is far beyond a simple power outage.

https://www.theregister.com/2023/11/02/cloudflare_outage/ """ In a nutshell, Cloudflare rolled out a new KV build to production. It turned out that the deployment tool had a bug, and some traffic got diverted to the wrong destination, which triggered a rollback … which failed. The result was that engineers had to manually switch the production route to the previous working version of Workers KV. The problem is that a…

The KV outage is the previous one, from Nov 1st.

We're currently in the Nov 2-3 outage, soon to rollover into Nov 4 in my timezone. This one is the power outage — also mentioned in the article — but unrelated to KV.

Re: Cloudflare outage – 24 hours now

#67

Post by joepie91_ » Sat, Nov 04, 2023, 12:08 AM UTC

olivermuty wrote: post #51

sailingparrot wrote: post #31

Earlier quoted context omitted.

We won't do it ourselves, but we also won't do it with a provider that has accumulated 50+ hours of downtime in less than a month all the while having no communication or support. That's barely clearing the one nine availability for the last 30 days (93%) for our particular stack on CF, this is insane. Mind you last time we were hit by a 22h outage on Oct. 9 we didn't get so much as an email from CF either during or…

To be fair, their status page says emails don’t work haha

It's long been accepted practice in the hosting industry to have your critical communications as a provider (status page, support system) hosted somewhere that's not your network, for this reason.

It continues to amaze me how major infrastructure providers seem to consistently fuck this one up (see also: AWS' status page outage a while ago).

Re: Cloudflare outage – 24 hours now

#68

Post by joepie91_ » Sat, Nov 04, 2023, 12:11 AM UTC

belthesar wrote: post #46

renonce wrote: post #36

Earlier quoted context omitted.

The Internet has been decentralized from the beginning. Now I don't want to claim that Cloudflare made something worse (at least it's enabling a lot of websites to exist without fear of DDoS) but the fact is that Cloudflare made it more centralized, as there are lots of websites that cannot be accessed without going through Cloudflare.

I think you might have missed the joke on this one.

Honestly, the original post could have been a joke, or it could not have been. I regularly talk to people who seem to genuinely believe this sort of thing (on this topic and others).

Re: Cloudflare outage – 24 hours now

#69

Post by sanat » Sat, Nov 04, 2023, 4:04 AM UTC

karlerss wrote: post #44

sanat wrote: post #12

I run hirevire.com one way video interview SaaS - and we were pretty much dead in the water during the Cloudflare Stream outage. We moved out to BunnyCDN's stream after waiting for 20 hours. One side benefit is that our videos are now stored in EU instead of Cloudflare's edge location around you.

How much work was the migration? Were the APIs feature-compatible or did you lose functionality?

The migration work was only a couple of hours for our core process. Took us 4 in total to restart collecting video.

We still have some accessory features to be moved to video on Bunny. Like transcriptions, downloads.

Re: Cloudflare outage – 24 hours now

#70

Post by ta1243 » Sat, Nov 04, 2023, 12:18 PM UTC

seized wrote: post #61

sangnoir wrote: post #48

Earlier quoted context omitted.

Not great to know an aspiring hyperscaler is one tornado away from insolvency.

I dont think PDX would get many tornados BUT the PNW has the joys of earthquakes that are overdue.

It's amusing, I have plans on how much service degradation I can accept and what I need to keep working in the UK if the Thames Barrier is breached (which would flood most of the UK's internet connectivity in places like Telehouse, Sovereign House, etc)

To have 30% of the internet relying on a single building in a single city is hilarious.