Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare was down

cloudflare.com

Post reply on HN
Print view View original

411–420 of 560 posts

Re: Cloudflare was down

#411

Post by mike_hearn »

koakuma-chan wrote: post #408
mike_hearn wrote: post #394

Earlier quoted context omitted.

Indeed, but fortunately there are more languages in the world than Rust and C++. A language that performed decently well and used exceptions systematically (Java, Kotlin, C#) would probably have recovered from a bad data file load.

There is nothing that prevents you from recovering from a bad data file load in Rust. The programmer who wrote that code chose to crash.

That's exactly my point. There should be no such thing as choosing to crash if you want reliable software. Choosing to crash is idiomatic in Rust but not in managed languages in which exceptions are the standard way to handle errors.

Re: Cloudflare was down

#412

Post by belter »

pm90 wrote: post #25

This is not good. One major outage? Something exceptional. Several outages in a short time? As someone thats worked in operations, I have empathy; there are so many “temp havks” that are put in place for incidents. but the rest of the world won’t… they’re gonna suffer a massive reputation loss if this goes on as long as the last one.

This will be another post-mortem of...config file messed...did not catch...promise to be doing better next....We are sorry.

They problem is architectural.

Re: Cloudflare was down

#413

Post by otikik »

agnivade wrote: post #348
berkes wrote: post #267

Earlier quoted context omitted.

But the nature of a CDN and most other products CF offers, is central by nature. If you switch from CF to the next CF competitor, you've not improved this dependency. The alternative here, is complex or even non-existing. Complex would be some system that allows you to hotswap a CDN, or to have fallback DDOS protection services, or to build you own in-house. Which, IMO, is the worst to do if your business is elsewher…

You can load-balance between CDN vendors as well

Then your load balancer becomes the single point of failure.

Re: Cloudflare was down

#414

Post by reassess_blind »

The "half the internet is down, nothing we can do" excuse works great the first time, but doesn't fly the second time in a month.

What solutions are there for Multi DNS/CDN failover that don't rely on a single point of failure?

Re: Cloudflare was down

#415

Post by da_grift_shift »

madjam002 wrote: post #340

Looking forward to the post mortem on this one. We weren't affected (just using the CDN), and people are saying they weren't affected who are using Cloudflare Workers (a previous culprit which we've since moved off), so I wonder what service / API was actually affected that brought down multiple websites with a 500 but not all of them. Wise was just down which is a pretty big one. Also odd how some websites were down…

The excuse:

>A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning.

>The change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components.

>We will share more information as we have it today.

https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q

Re: Cloudflare was down

#416

Post by phatfish »

coffeebeqn wrote: post #258
PlotCitizen wrote: post #96

Earlier quoted context omitted.

This is a good reminder for everyone to reconsider making all of their websites depend on a single centralized point of failure. There are many alternatives to the different services which Cloudflare offers.

We just love to merge the internet into single points of failure

This is just how free markets work, on the internet with no "physical" limitations it is simply accelerated.

Left alone corporations to rival governments emerge, which are completely unaccountable. At least there is some accountability of governments to the people, depending on your flavour of government.

Re: Cloudflare was down

#417

Post by mschuster91 »

coffeebeqn wrote: post #258
PlotCitizen wrote: post #96

Earlier quoted context omitted.

This is a good reminder for everyone to reconsider making all of their websites depend on a single centralized point of failure. There are many alternatives to the different services which Cloudflare offers.

We just love to merge the internet into single points of failure

no one loves the need for CDNs other than maybe video streaming services.

the problem is, below a certain scale you can't operate anything on the internet these days without hiding behind a WAF/CDN combo... with the cut-off mark being "we can afford a 24/7 ops team". even if you run a small niche forum no one cares about, all it takes is one disgruntled donghead that you ban to ruin the fun - ddos attacks are cheap and easy to get these days.

and on top of that comes the shodan skiddie crowd. some 0day pops up, chances are high someone WILL try it out in less than 60 minutes. hell, look into any web server log, the amount of blind guessing attacks (e.g. /wp-admin/..., /system/login, /user/login) or path traversal attempts is insane.

CDN/WAFs are a natural and inevitable outcome of our governments and regulatory agencies not giving a shit about internet security and punishing bad actors.

Re: Cloudflare was down

#420

Post by zelphirkalt »

berkes wrote: post #175
pm90 wrote: post #25

This is not good. One major outage? Something exceptional. Several outages in a short time? As someone thats worked in operations, I have empathy; there are so many “temp havks” that are put in place for incidents. but the rest of the world won’t… they’re gonna suffer a massive reputation loss if this goes on as long as the last one.

At least this warrants a good review of anyone's dependency on cloudflare. If it turns out that this was really just random bad luck, it shouldn't affect their reputation (if humans were rational, that is...) But if it is what many people seem to imply, that this is the outcome of internal problems/cuttings/restructuring/profit-increase etc, then I truly very much hope it affects their reputation. But I'm afraid it w…

Microsoft's reputation couldn't be much lower at this point, that's their trick.

The issue is the uninformed masses being led to use Windows when they buy a computer. They don't even know how much better a system could work, and so they accept whatever is shoved down their throats.

Post reply on HN