Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare was down

cloudflare.com

Post reply on HN
Print view View original

471–480 of 560 posts

Re: Cloudflare was down

#471

Post by rhdunn »

tialaramex wrote: post #459
philipwhiuk wrote: post #451

Earlier quoted context omitted.

So their parser broke again I guess. And no staged rollout I assume?

Apparently somehow this had never been how Cloudflare did this. I expressed incredulity about this to one of their employees, but yeah, seems like their attitude was "We never make mistakes so it's fastest to just deploy every change across the entire system immediately" and as we've seen repeatedly in the past short while that means it sometimes blows up. They have blameless post mortems, but maybe "We actually do m…

Blameless post mortems should be similar to air accident investigations. I.e. don't blame the people involved (unless they are acting maliciously), but identify and fix the issues to ensure this particular incident is unlikely to recur.

The intent of the postmortems is to learn what the issues are and prevent or mitigate similar issues happening in the future. If you don't make changes as a result of a postmortem then there's no point in conducting them.

Re: Cloudflare was down

#472

Post by bluerooibos »

pm90 wrote: post #25

This is not good. One major outage? Something exceptional. Several outages in a short time? As someone thats worked in operations, I have empathy; there are so many “temp havks” that are put in place for incidents. but the rest of the world won’t… they’re gonna suffer a massive reputation loss if this goes on as long as the last one.

I'm quite sure the reputational damage has already been done.

How do they not have better isolation of these issues, or redundancy of some sort?

Re: Cloudflare was down

#474

Post by Xunjin »

rhdunn wrote: post #471
tialaramex wrote: post #459

Earlier quoted context omitted.

Apparently somehow this had never been how Cloudflare did this. I expressed incredulity about this to one of their employees, but yeah, seems like their attitude was "We never make mistakes so it's fastest to just deploy every change across the entire system immediately" and as we've seen repeatedly in the past short while that means it sometimes blows up. They have blameless post mortems, but maybe "We actually do m…

Blameless post mortems should be similar to air accident investigations. I.e. don't blame the people involved (unless they are acting maliciously), but identify and fix the issues to ensure this particular incident is unlikely to recur. The intent of the postmortems is to learn what the issues are and prevent or mitigate similar issues happening in the future. If you don't make changes as a result of a postmortem the…

Agree 100%, however using your example, there is no regulatory agency that investigate the issue and demand changes to avoid related future problems. Should the industry move towards this way?

Re: Cloudflare was down

#475

Post by wooque »

pm90 wrote: post #25

This is not good. One major outage? Something exceptional. Several outages in a short time? As someone thats worked in operations, I have empathy; there are so many “temp havks” that are put in place for incidents. but the rest of the world won’t… they’re gonna suffer a massive reputation loss if this goes on as long as the last one.

2 days ago they had outage that affected Europe, Cloudflare seems to be going down the drain. I removed it for my personal sites.

Re: Cloudflare was down

#476

Post by wrobelda »

glimshe wrote: post #461

Not only they make my browsing experience a LOT worse (seconds per site for bot detection and additional "are you human" clicks even without VPNs), now they are bringing the entire Internet down. They don't deserve the position they currently have.

You make it sound like the DDoS and Bots are their fault.

Re: Cloudflare was down

#477

Post by gbrindisi »

berkes wrote: post #175
pm90 wrote: post #25

This is not good. One major outage? Something exceptional. Several outages in a short time? As someone thats worked in operations, I have empathy; there are so many “temp havks” that are put in place for incidents. but the rest of the world won’t… they’re gonna suffer a massive reputation loss if this goes on as long as the last one.

At least this warrants a good review of anyone's dependency on cloudflare. If it turns out that this was really just random bad luck, it shouldn't affect their reputation (if humans were rational, that is...) But if it is what many people seem to imply, that this is the outcome of internal problems/cuttings/restructuring/profit-increase etc, then I truly very much hope it affects their reputation. But I'm afraid it w…

The crowdstrike incident taught us that no one is going to review any dependency whatsoever.

Re: Cloudflare was down

#478

Post by [deleted] »

mike_hearn wrote: post #411
koakuma-chan wrote: post #408

Earlier quoted context omitted.

There is nothing that prevents you from recovering from a bad data file load in Rust. The programmer who wrote that code chose to crash.

That's exactly my point. There should be no such thing as choosing to crash if you want reliable software. Choosing to crash is idiomatic in Rust but not in managed languages in which exceptions are the standard way to handle errors.

[deleted]

Re: Cloudflare was down

#479

Post by aatd86 »

luastoned wrote: post #439

From the incident page: A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today. https://www.cloudflarestatus.com…

so it's react again in the end .. zzzzzzz

Re: Cloudflare was down

#480

Post by lucianbr »

netdevphoenix wrote: post #333
lucianbr wrote: post #288

Earlier quoted context omitted.

Are the contracts so easy to bypass? Who signs a contract with an SLA knowing the service provider will just lie about the availability? Is the client supposed to sue the provider any time there is an SLA breach?

Anyone who doesn't have any choice financially or gnostically. Same reason why people pay Netflix despite the low quality of most of their shows and the constant termination of tv series after 1 season. Same reason why people put up with Meta not caring about moderating or harmful content. The power dynamics resemble a monopoly

Why bother to put the SLA in the contract at all, if people have no choice but to sign it?

Netflix doesn't put in the contract that they will have high-quality shows. (I guess, don't have a contract to read right now.)

Post reply on HN