Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare was down

cloudflare.com

Post reply on HN
Print view View original

491–500 of 560 posts

Re: Cloudflare was down

#491

Post by kypro »

tialaramex wrote: post #459
philipwhiuk wrote: post #451

Earlier quoted context omitted.

So their parser broke again I guess. And no staged rollout I assume?

Apparently somehow this had never been how Cloudflare did this. I expressed incredulity about this to one of their employees, but yeah, seems like their attitude was "We never make mistakes so it's fastest to just deploy every change across the entire system immediately" and as we've seen repeatedly in the past short while that means it sometimes blows up. They have blameless post mortems, but maybe "We actually do m…

> They have blameless post mortems, but maybe "We actually do make mistakes so this practice is not good" wasn't a lesson anybody wanted to hear.

Or they could say, "we want to continue to prioritise speed of security rollouts over stability, and despite our best efforts, we do make mistakes, so sometimes we expect things will blow up".

I guess it depends what you're optimising for... If the rollout speed of security patches is the priority then maybe increased downtime is a price worth paying (in their eyes anyway)... I don't agree with that, but at least it's an honest position to take.

That said, if this was to address the React CVE then it was hardly a speedy patch anyway... You'd think they could have afforded to stagger the rollout over a few hours at least.

Re: Cloudflare was down

#493

Post by tehlike »

lima wrote: post #486
reassess_blind wrote: post #458

Earlier quoted context omitted.

I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.

They don't appear to have a rollout procedure for some of their globally replicated application state. They had a number of major outages over the past years which all had the same root cause of "a global config change exposed a bug in our code and everything blew up". I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do th…

You can selectively bypass many roll out procedures in a properly designed system.

Re: Cloudflare was down

#494

Post by gilrain »

glimshe wrote: post #461

Not only they make my browsing experience a LOT worse (seconds per site for bot detection and additional "are you human" clicks even without VPNs), now they are bringing the entire Internet down. They don't deserve the position they currently have.

> Not only they make my browsing experience a LOT worse

No, I did (metaphorically, for the websites I control). And I did it because otherwise those sites are fully offline or unusable thanks to the modern floods of unfilterable scrapers.

Months of piecemeal mitigations, but Attack Mode is the only thing that worked. Blame the LLM gold rush and the many, many software engineers with no ethics and zero qualms about racing to find the bottom of the Internet.

Re: Cloudflare was down

#495

Post by meindnoch »

rhdunn wrote: post #471
tialaramex wrote: post #459

Earlier quoted context omitted.

Apparently somehow this had never been how Cloudflare did this. I expressed incredulity about this to one of their employees, but yeah, seems like their attitude was "We never make mistakes so it's fastest to just deploy every change across the entire system immediately" and as we've seen repeatedly in the past short while that means it sometimes blows up. They have blameless post mortems, but maybe "We actually do m…

Blameless post mortems should be similar to air accident investigations. I.e. don't blame the people involved (unless they are acting maliciously), but identify and fix the issues to ensure this particular incident is unlikely to recur. The intent of the postmortems is to learn what the issues are and prevent or mitigate similar issues happening in the future. If you don't make changes as a result of a postmortem the…

>don't blame the people involved (unless they are acting maliciously)

Or negligently.

Re: Cloudflare was down

#496

Post by glimshe »

wrobelda wrote: post #476
glimshe wrote: post #461

Not only they make my browsing experience a LOT worse (seconds per site for bot detection and additional "are you human" clicks even without VPNs), now they are bringing the entire Internet down. They don't deserve the position they currently have.

You make it sound like the DDoS and Bots are their fault.

They make gazillions. I'm sure they can do better than that.

How many awful things in tech can be rationalized away by "sorry, but this is for you/our protection"?

Re: Cloudflare was down

#497

Post by lima »

tehlike wrote: post #493
lima wrote: post #486

Earlier quoted context omitted.

They don't appear to have a rollout procedure for some of their globally replicated application state. They had a number of major outages over the past years which all had the same root cause of "a global config change exposed a bug in our code and everything blew up". I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do th…

You can selectively bypass many roll out procedures in a properly designed system.

If there is a proper rollout procedure that would've caught this, and they bypass it for routine WAF configuration changes, they might as well not have one.

Re: Cloudflare was down

#498

Post by GoblinSlayer »

koakuma-chan wrote: post #431
mike_hearn wrote: post #411

Earlier quoted context omitted.

That's exactly my point. There should be no such thing as choosing to crash if you want reliable software. Choosing to crash is idiomatic in Rust but not in managed languages in which exceptions are the standard way to handle errors.

I am not a C# guy, but I wrote a lot of Java back in the day, and I can authoritatively tell you that it has so-called "checked exceptions" that the compiler forces you to handle. However, it also has "runtime exceptions" that you are not forced to handle, and they can happen any where and any time. Conceptually, it is the same as error versus panic in Rust. One such runtime exception is the notorious `java.lang.Null…

When dotnet has an unhandled exception, it terminates with abort.

Re: Cloudflare was down

#499

Post by spiffytech »

meindnoch wrote: post #456
luastoned wrote: post #439

From the incident page: A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today. https://www.cloudflarestatus.com…

React (a frontend JS framework) can now bring down critical Internet infrastructure. I will repeat it because it's so surreal: React (a frontend JS framework) can now bring down critical Internet infrastructure.

To be accurate: React developed server-side capabilities, and that's where the vulnerability exists.

It's feels noteworthy because React started out frontend-only, but pedantically it's just another backend with a vulnerability.

Re: Cloudflare was down

#500

Post by [deleted] »

stogot wrote: post #490
reassess_blind wrote: post #458

Earlier quoted context omitted.

I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.

The update they describe should never bring down all services. I agree with other posters that they must lack a rollout strategy yet they sent spam emails mocking the reliability of other clouds

[deleted]
Post reply on HN