Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

771–780 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#771

Post by bombela »

StopDisinfo910 wrote: post #523
bombela wrote: post #490

Earlier quoted context omitted.

This thread warms my heart. Rust has set a new baseline that many and myself now take for granted. We are now discussing what can be done to improve code correctness beyond memory and thread safety. I am excited for what is to come.

Alternatively you can look at actually innovative programming languages to peek at the next 20 years of innovation. I am not sure that watching the trendy forefront successfully reach the 1990s and discuss how unwrapping Option is potentially dangerous really warm my heart. I can’t wait for the complete meltdown when they discover effect systems in 2040. To be more serious, this kind of incident is yet another remind…

I agree with you. But onsidering nobody learns any real engineering in software; myself solidly included; this is still an improvement.

But yes, I wish I had learned more, and somehow stumbled upon all the good stuff, or be taught at university about at least what Rust achieves today.

I think it has to be noted Rust still allows performance with the safety it provides. So that's something maybe.

Re: Cloudflare outage on November 18, 2025 post mortem

#773

Post by brookst »

cowsandmilk wrote: post #577
polack wrote: post #570

Earlier quoted context omitted.

They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…

> Why do they do non-critical changes in production before testing in a stage environment? I guess the noncritical change here was the change to the database? My experience has been a lot of teams do a poor job having a faithful replica of databases in stage environments to expose this type of issue.

In part because it is somewhere between really hard and impossible. Is your staging DB going to be as big? Seeing the same RPS as prod? Seeing the same scenarios?

Permissions stuff might be caught without a completely faithful replica, but there are always going to be attributes of the system that only exist in prod.

Re: Cloudflare outage on November 18, 2025 post mortem

#774

Post by Ygg2 »

antihero wrote: post #694
Ygg2 wrote: post #541

Earlier quoted context omitted.

> If the `.unwrap()` was replaced with `.expect("Feature config is too large!")` it would certainly make the outage shorter. It wouldn't, not meaningfully. The outage was caused by change in how they processed the queries. They had no way to observe the changes, nor canaries to see that change is killing them. Plus, they would still need to manually feed and restart services that ingested bad configs. `expect` would…

A billion alerts in DD/Sentry/whatever saying the exact problem that coincide with the exact graph of failures would probably be helpful if someone looked at them.

Not if everyone decides to 's/.unwrap()/.expect("Shouldn't happen")/g' in the code base.

Or the good old:

    let x = match res { 
       Ok(x) => x,
       Err(_) => unreachable!(),
    }

Re: Cloudflare outage on November 18, 2025 post mortem

#775

Post by tempest_ »

throw7 wrote: post #772

Is this true: from that core proxy diagram, I didn't realize cloudflare sees the full unencrypted packet between you and the server. If that's true, is there a way to tell (easily) whether a site is using cloudflare or not?

It is pretty easy to see if cloudflare is proxying a site.

Just ping the host and see if the ip belongs to CF.

https://www.cloudflare.com/en-ca/ips/

Re: Cloudflare outage on November 18, 2025 post mortem

#776

Post by gempir »

reassess_blind wrote: post #94
eastdakota wrote: post #32

Earlier quoted context omitted.

We don’t know. Suspect it may just have been a big uptick in load and a failure of its underlying infrastructure to scale up.

The status page is hosted on AWS Cloudfront, right? It sure looks like Cloudfront was overwhelmed by the traffic spike, which is a bit concerning. Hope we'll see a post from their side.

CloudFront has quotas[0] and they likely just hit those quota limits. To request higher quotas requires a service ticket. If they have access logs enabled in CloudFront they could see what the exact error was.

And since it seems this is hosted by Atlassian, this would be up to Atlassian.

[0] https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...

Re: Cloudflare outage on November 18, 2025 post mortem

#777

Post by finally7394 »

throw7 wrote: post #772

Is this true: from that core proxy diagram, I didn't realize cloudflare sees the full unencrypted packet between you and the server. If that's true, is there a way to tell (easily) whether a site is using cloudflare or not?

The NSA has to see the data somehow, right?

Re: Cloudflare outage on November 18, 2025 post mortem

#778

Post by deadcore »

anonymous908213 wrote: post #597
deadcore wrote: post #526

Earlier quoted context omitted.

Out of interest... apart from the em dash, how else can you tell it's an LLM response? What are the telltales signs?

This particular excerpt is reeking of it with pretty much every line. I'll point out the patterns in the English translation, but all of these patterns apply cross-language. > classic chain reaction triggered by “hidden assumptions + configuration chains” "Classic/typical "x + y"", particularly when diagnosing an issue. This one is a really easy tell because humans, on aggregate, do not use quotation marks like this.…

Damn brother - thanks for this!

Re: Cloudflare outage on November 18, 2025 post mortem

#779

Post by smt88 »

vlaaad wrote: post #634
smt88 wrote: post #392

Earlier quoted context omitted.

Cloudflare, Azure, and other single points of failure are solving issues inherent to webhosting, and those problems have become incredibly hard due to the massive scale of bad actors and the massive complexity of managing hardware and software. What would you propose to fix it? The fixed cost of being DDoS-proof is in the hundreds of millions of dollars.

Saying "other single points of failure" makes no sense whatsoever.

Cloudflare and us-east-1 are single points of failures for millions of their customers, many of which overlap.

"Single" means "you only need one," not that there is only one.

Re: Cloudflare outage on November 18, 2025 post mortem

#780

Post by branko_d »

frumplestlatz wrote: post #556
branko_d wrote: post #469

Earlier quoted context omitted.

> Exceptions force a panic on all errors What do you mean? Exceptions do not force panic at all. In most practical situations, an exception unhandled close to where it was thrown will eventually get logged. It's kind of a "local" panic, if you will, that will terminate the specific function, but the rest of the program will remain unaffected. For example, a web server might throw an exception while processing a speci…

> Throwing an exception does not necessarily mean that your program is suddenly in an unsupported state When everyone uses runtime exceptions and doesn’t count for exception handling in every possible code path, that’s exactly what it means.

Sure, but the same is true of any error handling strategy.

When you work with exceptions, the key is to assume that every line can throw unless proven otherwise, which in practice means almost all lines of code can throw. Once you adopt that mental model, things get easier.

Post reply on HN