Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

231–240 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#231

Post by dehrmann »

wrs wrote: post #86
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic. It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used ca…

It's the same blind spot people have to Java's checked exceptions. People commonly resort to Pokemon exception handling and either blindly ignoring or rethrowing as a runtime exception. When Rust got popular, I was a bit confused by people talking about how great Result it's essentially a checked exception without a stack trace.

Re: Cloudflare outage on November 18, 2025 post mortem

#232

Post by duped »

dietr1ch wrote: post #123
SchwKatze wrote: post #111

Earlier quoted context omitted.

Unwrap isn't a synonym for laziness, it's just like an assertion, when you do unwrap() you're saying the Result should NEVER fail, and if does, it should abort the whole process. What was wrong was the developer assumption, not the use of unwrap.

> What was wrong was the developer assumption, not the use of unwrap. How many times can you truly prove that an `unwrap()` is correct and that you also need that performance edge? Ignoring the performance aspect that often comes from a hat-trick, to prove such a thing you need to be wary of the inner workings of a call giving you a `Return`. That knowledge is only valid at the time of writing your `unwrap()`, but wo…

Panics are for unexpected error conditions, like your caller passed you garbage. Results are for expected errors, like your caller passed you something but it's your job to tell if it's garbage.

So the point of unwrap() is not to prove anything. Like an assertion it indicates a precondition of the function that the implementer cannot uphold. That's not to say unwrap() can't be used incorrectly. Just that it's a valid thing to do in your code.

Note that none of this is about performance.

Re: Cloudflare outage on November 18, 2025 post mortem

#233

Post by ashishb »

tristan-morris wrote: post #60
ashishb wrote: post #53

Earlier quoted context omitted.

> You are saying this would not have happened in a C release build where asserts define to nothing? Afaik, Go and Java are the only languages that make you pause and explicitly deal with these exceptions.

And rust, but they chose to panic on the error condition. Wild.

> And rust, but they chose to panic on the error condition. Wild.

unwrap() implicitly panic-ed, right?

Re: Cloudflare outage on November 18, 2025 post mortem

#234

Post by JuniperMesos »

wildmXranat wrote: post #138

Hold up ,- when I used a C or similar language for accessing a database and wanted to clamp down on memory usage to deterministically control how much I want to allocated, I would explicitly limit the number of rows in the query. There never was an unbound "select all rows from some table" without a "fetch first N rows only" or "limit N" If you knew that this design is rigid, why not leverage the query to actually do…

Because nothing forced them to and they didn't think of it. Maybe the people writing the code that did the query knew that the tables they were working with never had more than 60 rows and figured "that's small" so they didn't bother with a limit. Maybe the people who wrote the file size limit thought "60 rows isn't that much data" and made a very small file size limit and didn't coordinate with the first people.

Anyway regardless of which language you use to construct a SQL query, you're not obligated to put in a max rows

Re: Cloudflare outage on November 18, 2025 post mortem

#235

Post by otterley »

HumanOstrich wrote: post #226
otterley wrote: post #180

Earlier quoted context omitted.

Suppose they did have the cellular architecture today, but every other fact was identical. They'd still have suffered the failure! But it would have been contained , and the damage would have been far less. Fires happen every day. Smoke alarms go off, firefighters get called in, incident response is exercised, and lessons from the situation are learned (with resulting updates to the fire and building codes). Yet even…

What variant of cellular architecture are you referring to? Can you give me a link or few? I'm fascinated by it and I've led a team to break up a monolithic solution running on AWS to a cellular architecture. The results were good, but not magic. The process of learning from failures did not stop, but it did change (for the better). No matter what architecture, processes, software, frameworks, and systems you use, or…

If your AWS service is properly regionalized, that’s the minimum amount of cellular architecture required. Did your service ever fail in multiple regions simultaneously?

Cellular architecture within a region is the next level and is more difficult, but is achievable if you adhere to the same principles that prohibit inter-regional coupling:

https://docs.aws.amazon.com/wellarchitected/latest/reducing-...

https://docs.aws.amazon.com/wellarchitected/latest/reducing-...

Re: Cloudflare outage on November 18, 2025 post mortem

#236

Post by pdimitar »

kloop wrote: post #192
SchemaLoad wrote: post #131

Earlier quoted context omitted.

It also makes it very obvious in the code, something very dangerous is happening here. As a code reviewer you should see an unwrap() and have alarm bells going off. While in other languages, critical errors are a lot more hidden.

I hate that it's a method. That can get lost in a method chain easily enough during a code review. A function or a keyword would interrupt that and make it less tempting

Well, you can request Clippy to tell you about them. I do that in my hobby projects.

Re: Cloudflare outage on November 18, 2025 post mortem

#238

Post by wrs »

tux3 wrote: post #165
dist1ll wrote: post #149

Earlier quoted context omitted.

> every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used can enforce this. How about indexing into a slice/map/vec? Should every `foo[i]` have an infallibility comment? Because they're essentially `get(i).unwrap()`.

Usually you'd want to write almost all your slice or other container iterations with iterators, in a functional style. For the 5% of cases that are too complex for standard iterators? I never bother justifying why my indexes are correct, but I don't see why not. You very rarely need SAFETY comments in Rust because almost all the code you write is safe in the first place. The language also gives you the tool to avoid…

I didn't restate the context from the code we're discussing: it must not panic. If you don't care if the code panics, then go ahead and unwrap/expect/index, because that conforms to your chosen error handling scheme. This is fine for lots of things like CLI tools or isolated subprocesses, and makes review a lot easier.

So: first, identify code that cannot be allowed to panic. Within that code, yes, in the rare case that you use [i], you need to at least try to justify why you think it'll be in bounds. But it would be better not to.

There are a couple of attempts at getting the compiler to prove that code can't panic (e.g., the no-panic crate).

Re: Cloudflare outage on November 18, 2025 post mortem

#239

Post by JuniperMesos »

back_to_basics wrote: post #173

While it's certainly worthwhile to discuss the Technical and Procedural elements that contributed to this Service Outage, the far more important (and mutually-exclusive aspect) to discuss should be: Why have we built / permitted the building of / Subscribed to such a Failure-intolerant "Network"?

Who's "we"? This is not a trick question, what specific people do you think acted wrongly here? I don't use Cloudflare personally. I don't run any of the sites that do use it. The people who did make the decision to put thier websites behind Cloudflare could stop, and maybe some will, but presumably they're paying for it because they think, perhaps accurately, that they get value out of it. Should some power compel them not to use Cloudflare?

Re: Cloudflare outage on November 18, 2025 post mortem

#240

Post by 9029 »

cherryteastain wrote: post #193
gpm wrote: post #181

Earlier quoted context omitted.

Unwrap is used in places where in C++ you would just have undefined behavior. It wouldn't make any more sense to blanket ban it than it would ban ever dereferencing a pointer just in case its null - even if you just checked that it wasn't null.

Rust's Result is the same thing as C++'s std::expected. How is calling std::expected::value undefined behaviour?

Tangential but funnily enough calling std::expected::error is ub if there is no error :D
Post reply on HN