Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

551–560 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#551

Post by frumplestlatz »

StopDisinfo910 wrote: post #523
bombela wrote: post #490

Earlier quoted context omitted.

This thread warms my heart. Rust has set a new baseline that many and myself now take for granted. We are now discussing what can be done to improve code correctness beyond memory and thread safety. I am excited for what is to come.

Alternatively you can look at actually innovative programming languages to peek at the next 20 years of innovation. I am not sure that watching the trendy forefront successfully reach the 1990s and discuss how unwrapping Option is potentially dangerous really warm my heart. I can’t wait for the complete meltdown when they discover effect systems in 2040. To be more serious, this kind of incident is yet another remind…

I find myself in the odd position of agreeing with you both.

That we’re even having this discussion is a major step forward. That we’re still having this discussion is a depressing testament to how slow slowly the mainstream has adopted better ideas.

Re: Cloudflare outage on November 18, 2025 post mortem

#552

Post by wongarsu »

brabel wrote: post #495
wrs wrote: post #86

Earlier quoted context omitted.

It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic. It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used ca…

Yes, I always thought it was wrong to use unwrap in examples. I know, people want to keep examples simple, but it trains developers to use unwrap() as they see that everywhere. Yes, there are places where it's ok as that blog post explains so well: https://burntsushi.net/unwrap/ But most devs IMHO don't have the time to make the call correctly most of the time... so it's just better to do something better, like handl…

> at least do `expect("damn it, how did this happen")`

That gives you the same behavior as unwrap with a less useful error message though. In theory you can write useful messages, but in practice (and your example) expect is rarely better than unwrap in modern rust

Re: Cloudflare outage on November 18, 2025 post mortem

#553

Post by the8472 »

dzonga wrote: post #54

> thread fl2_worker_thread panicked: called Result::unwrap() on an Err value I don't use Rust, but a lot of Rust people say if it compiles it runs. Well Rust won't save you from the usual programming mistake. Not blaming anyone at cloudflare here. I love Cloudflare and the awesome tools they put out. end of day - let's pick languages | tech because of what we love to do. if you love Rust - pick it all day. I actually…

"if it compiles it runs" - this is indeed an inaccurate marketing slogan. A more precise formulation would be "if it compiles then the static type system, pattern matching, explicit errors, Send bounds, etc. will have caught a lot of bugs that in other languages would have manifested as runtime errors".

Anecdotally I can write code for several hours, deploy it to a test sandbox without review or running tests and it will run well enough to use it, without silly errors like null pointer exceptions, type mismatches, OOBs etc. That doesn't mean it's bug-free. But it doesn't immediately crash and burn either. Recently I even introduced a bug that I didn't immediately notice because careful error handling in another place recovered from it.

Re: Cloudflare outage on November 18, 2025 post mortem

#554

Post by zelphirkalt »

abalone wrote: post #380
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

It is just 2 different layers. Of course the code is also a problem, if it is in fact as the GP describes it. You are taking the higher level view, which is the second layer of dealing with not only this specific mistake, but also other mistakes, that can be related to arbitrary code paths.

Both are important, and I am pretty sure, that someone is gonna fix that line of code pretty soon.

Re: Cloudflare outage on November 18, 2025 post mortem

#555

Post by frumplestlatz »

never_inline wrote: post #377
bigstrat2003 wrote: post #315

Earlier quoted context omitted.

I'm with you! Checked exceptions are actually good and the hate for them is super short sighted. The exact same criticisms levied at checked exceptions apply to static typing in general, but people acknowledge the great value static types have for preventing errors at compile time. Checked exceptions have that same value, but are dunked on for some reason.

The dislike is probably because of 2 reasons. 1. in most cases they don't want to handle `InterruptedException` or `IOException` and yet need to bubble them up. In that case the code is very verbose. 2. it makes lambdas and functions incompatible. So eg: if you're passing a function to forEach, you're forced to wrap it in runtime exception. 3. Due to (1) and (2), most people become lazy and do `throws Exception` whic…

> an uncaught exception is not a big deal

In my experience, it actually is a big deal, leaving a wake of indeterminant state behind after stack unrolling. The app then fails with heisenbugs later, raising more exceptions that get ignored, compounding the problem.

People just shrug off that unreliability as an unavoidable cost of doing business.

Re: Cloudflare outage on November 18, 2025 post mortem

#556

Post by frumplestlatz »

branko_d wrote: post #469
gwbas1c wrote: post #281

Earlier quoted context omitted.

It's a lot lighter: a stack trace takes a lot of overhead to generate; a result has no overhead for a failure. The overhead (panic) only comes once the failure can't be handled. (Most books on Java/C# don't explain that throwing exceptions has high performance overhead.) Exceptions force a panic on all errors, which is why they're supposed to be used in "exceptional" situations. To avoid exceptions when an error is e…

> Exceptions force a panic on all errors What do you mean? Exceptions do not force panic at all. In most practical situations, an exception unhandled close to where it was thrown will eventually get logged. It's kind of a "local" panic, if you will, that will terminate the specific function, but the rest of the program will remain unaffected. For example, a web server might throw an exception while processing a speci…

> Throwing an exception does not necessarily mean that your program is suddenly in an unsupported state

When everyone uses runtime exceptions and doesn’t count for exception handling in every possible code path, that’s exactly what it means.

Re: Cloudflare outage on November 18, 2025 post mortem

#557

Post by frumplestlatz »

baq wrote: post #505
dehrmann wrote: post #231

Earlier quoted context omitted.

It's the same blind spot people have to Java's checked exceptions. People commonly resort to Pokemon exception handling and either blindly ignoring or rethrowing as a runtime exception. When Rust got popular, I was a bit confused by people talking about how great Result it's essentially a checked exception without a stack trace.

checked exceptions failed because when used properly they fossilize method signatures. they're fine if your code will never be changed and they're fine when you control 100% of users of the throwing code. if you're distributing a library... no bueno.

That’s just not true. They required that you use hierarchical exception types and define your own library exception type that you declare at the boundary.

The same is required for any principled error handling.

Re: Cloudflare outage on November 18, 2025 post mortem

#558

Post by ErikCorry »

dehrmann wrote: post #454
never_inline wrote: post #377

Earlier quoted context omitted.

The dislike is probably because of 2 reasons. 1. in most cases they don't want to handle `InterruptedException` or `IOException` and yet need to bubble them up. In that case the code is very verbose. 2. it makes lambdas and functions incompatible. So eg: if you're passing a function to forEach, you're forced to wrap it in runtime exception. 3. Due to (1) and (2), most people become lazy and do `throws Exception` whic…

> 2. it makes lambdas and functions incompatible. This is true, but the hate predated lambdas in Java.

You could always manually build the same thing as lambda with a class and you had the same problem.

Re: Cloudflare outage on November 18, 2025 post mortem

#559

Post by ithkuil »

nrhrjrjrjtntbt wrote: post #496
abalone wrote: post #380

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

This guy SREs

Back when it meant Site Reliability Engineer and not Sysadmin Really Expensive

Re: Cloudflare outage on November 18, 2025 post mortem

#560

Post by zelphirkalt »

koito17 wrote: post #482
dist1ll wrote: post #225

Earlier quoted context omitted.

If that's the case then hats off. What you're describing is definitely not what I've seen in practice. In fact, I don't think I've ever seen a crate or production codebase that documents infallibility of every single slice access. Even security-critical cryptography crates that passed audits don't do that. Personally, I found it quite hard to avoid indexing for graph-heavy code, so I'm always on the lookout for inter…

> I don't think I've ever seen a crate or production codebase that documents infallibility of every single slice access. The smoltcp crate typically uses runtime checks to ensure slice accesses made by the library do not cause a panic. It's not exactly equivalent to GP's assertion, since it doesn't cover "every single slice access", but it at least covers slice accesses triggered by the library's public API. (i.e. no…

I think this goes against the Rust goals in terms of performance. Good for safe code, of course, but usually Rust users like to have compile time safety to making runtime safety checks unnecessary.
Post reply on HN