Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare was down

cloudflare.com

Post reply on HN
Print view View original

441–450 of 560 posts

Re: Cloudflare was down

#441

Post by iso1631 »

phartenfeller wrote: post #402
8cvor6j844qw_d6 wrote: post #300

Earlier quoted context omitted.

I'll need to checkup on DigitalOcean uptime, may be better than Cloudflare.

My Hetzner servers have been running fine for years. Okay, there were times when I broke something, but at least I was able to fix it quickly and never felt dependent on others.

CxOs want to be dependent on someone else, specifically suppliers with pieces of paper saying "we are great, here's a 1% discount on next years renewal"

If the in house tech team breaks something and fixes it, that's great from an engineer point of view - we like to be useful, but the person at the top is blamed.

If an outsourced supplier (one which the consultants recommend, look at Gartner Quadrants etc) fails, then the person at the top is not blamed, even though they are powerless and the outage is 10 times longer and 10 times as frequent.

Outsourcing is not about outcome, it's about accountability, and specifically avoiding it.

Re: Cloudflare was down

#442

Post by GoblinSlayer »

da_grift_shift wrote: post #415
madjam002 wrote: post #340

Looking forward to the post mortem on this one. We weren't affected (just using the CDN), and people are saying they weren't affected who are using Cloudflare Workers (a previous culprit which we've since moved off), so I wonder what service / API was actually affected that brought down multiple websites with a 500 but not all of them. Wise was just down which is a pretty big one. Also odd how some websites were down…

The excuse: >A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. >The change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. >We will share more information as we have it today. https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q

Who sent an xml request?

Re: Cloudflare was down

#443

Post by inferiorhuman »

everfrustrated wrote: post #422
esseph wrote: post #374

Earlier quoted context omitted.

> The culture clearly is not fit for their level of criticality I don't think anyone's is.

How often do you hear of Akamai going down and they host a LOT more enterprise/high value sites than Cloudflare. There's a reason Cloudflare has been really struggling to get into the traditional enterprise space and it isn't price.

A quick google turned up an Akamai outage in July that took Linode down and two in 2021. At that scale nobody's going to come up smelling like roses. I mostly dealt with Amazon crap at megacorp, but nobody that had to deal with our Akamai stuff had anything kind to say about them as a vendor.

At first blush it's getting harder to "defend" use of Cloudflare, but I'll wait until we get some idea of what actually broke. For the time being I'll save my outrage for the AI scrapers that drove everyone into Cloudflare's arms.

Re: Cloudflare was down

#444

Post by rsynnott »

berkes wrote: post #175
pm90 wrote: post #25

This is not good. One major outage? Something exceptional. Several outages in a short time? As someone thats worked in operations, I have empathy; there are so many “temp havks” that are put in place for incidents. but the rest of the world won’t… they’re gonna suffer a massive reputation loss if this goes on as long as the last one.

At least this warrants a good review of anyone's dependency on cloudflare. If it turns out that this was really just random bad luck, it shouldn't affect their reputation (if humans were rational, that is...) But if it is what many people seem to imply, that this is the outcome of internal problems/cuttings/restructuring/profit-increase etc, then I truly very much hope it affects their reputation. But I'm afraid it w…

> Just like Microsoft continues to push out software, that, compared to competitors, is unstable, insecure, frustrating to use, lacks features, etc, without it harming their reputation or even bottomlines too much.

Eh.... This is _kind_ of a counterfactual, tho. Like, we are not living in the world where MS did not do that. You could argue that MS was in a good place to be the dominant server and mobile OS vendor, and simply screwed both up through poor planning, poor execution, and (particularly in the case of server stuff) a complete disregard for quality as a concept.

I think someone who'd been in a coma since 1999 waking up today would be baffled at how diminished MS is, tbh. In the late 90s, Microsoft practically _was_ computers, with only a bunch of mostly-dying UNIX vendors for competition. And one reasonable lens through which to interpret its current position is that it's basically due to incompetence on Microsoft's part.

Re: Cloudflare was down

#445

Post by jcmfernandes »

pm90 wrote: post #25

This is not good. One major outage? Something exceptional. Several outages in a short time? As someone thats worked in operations, I have empathy; there are so many “temp havks” that are put in place for incidents. but the rest of the world won’t… they’re gonna suffer a massive reputation loss if this goes on as long as the last one.

Absolutely. I wouldn’t be surprised if they turned the heat up a little after the last incident. The result? Even more incidents.

Re: Cloudflare was down

#446

Post by PhilippGille »

GoblinSlayer wrote: post #289
javier2 wrote: post #112

Earlier quoted context omitted.

Yeah. I only work for a small company, but you can be certain we will not update the status page if only a small portion of customers are affected, and if we are fully down, rest assured there will be no available hands to keep the status page updated

You won't be able to update the status page due to failures anyway.

Why not? A good status page runs on a different cloud provider in a different region, specifically to not be affected at the same time.

Re: Cloudflare was down

#447

Post by iso1631 »

yoctosec wrote: post #382
runeb wrote: post #202

Earlier quoted context omitted.

Checkout tailscale

And what about a website I want to make public? I'm just concerned about my IP being visible, like for my personal website or my searxng instance

Personally I'd just proxy it through a vm running on hertzer, linode, rackspace, etc

Re: Cloudflare was down

#449

Post by inferiorhuman »

PlotCitizen wrote: post #96
pm90 wrote: post #25

This is not good. One major outage? Something exceptional. Several outages in a short time? As someone thats worked in operations, I have empathy; there are so many “temp havks” that are put in place for incidents. but the rest of the world won’t… they’re gonna suffer a massive reputation loss if this goes on as long as the last one.

This is a good reminder for everyone to reconsider making all of their websites depend on a single centralized point of failure. There are many alternatives to the different services which Cloudflare offers.

  There are many alternatives
Of varying quality depending on the service. Most of the anti-bot/catpcha crap seems to be equivalently obnoxious, but the handful of sites that use PerimeterX… I've basically sworn off DigiKey as a vendor since I keep getting their bullshit "press and hold" nonsense even while logged in.

I don't like that we're trending towards a centralized internet, but that's where we are.

Re: Cloudflare was down

#450

Post by asim »

I'd like to start seeing the architecture and design of how cloudflare works. Not blog posts, like a whole write-up. If you're going to have this many outages and you're a public company which 2/3rd of US infrastructure probably depends on then it might need some external input. Obviously they know what they're doing. This is not a blame game but the tools are starting to creak.
Post reply on HN