Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

451–460 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#451

Post by jeffrallen »

eastdakota wrote: post #95
yen223 wrote: post #57

Earlier quoted context omitted.

I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency. Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.

Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current…

A postmortem postmortem, I love it. Transparency to the power of 2.

Re: Cloudflare outage on November 18, 2025 post mortem

#452

Post by aw1621107 »

tonyhart7 wrote: post #430
aw1621107 wrote: post #385

Earlier quoted context omitted.

> because these thing called BEST MOVE and BAD MOVE there in chess The thing is that there is no known general objective criteria for "best" and "bad" moves. The best we have so far is based on engine evaluations, but as I said before that is because chess engines are better at searching the board's state space than humans, not because chess engines have solved chess in the mathematical sense. Engines are quite capab…

"The thing is that there is no known general objective criteria for "best" and "bad" moves." are you playing chess or not?????? if you playing chess then its oblivious how to differentiate bad move and best move Yes it is objective, these thing called best move not without reason "If you think chess engines are infalliable, then why does the Top Chess Engine Championship exist?" to create better chess engine like wha…

> if you playing chess then its oblivious how to differentiate bad move and best move

The key words in what I said are "general" and "objective". Yes, it's possible to determine "good" or "bad" moves in specific positions. There's no known method to determine "good" or "bad" moves in arbitrary positions, as would be required for chess to be considered strongly solved.

Furthermore, if it's "obvious" how to differentiate good and bad moves then we should never see engines blundering, right?

So (for example) how do you explain this game between Stockfish and Leela where Stockfish blunders a seemingly winning position [0]? After 37... Rdd8 both Stockfish and Leela think white is clearly winning (Stockfish's evaluation is +4.00, while Leela's evaluation is +3.81), but after 38. Nxb5 Leela's evaluation plummets to +0.34 while Stockfish's evaluation remains at +4.00. In the end, it turns out Leela was correct after 40... Rxc6 Stockfish's evaluation also drops from +4.28 to 0.00 as it realizes that Leela has a forced stalemate.

Or this game also between Stockfish and Leela where Leela blunders into a forced mating sequence and doesn't even realize it for a few moves [1]?

Engines will presumably always play what they think is the "best" move, but clearly sometimes this "best" move is wrong. Evidently, this means differentiating "good" and "bad" moves is not always obvious.

> Yes it is objective, these thing called best move not without reason

If it's objective, then why is it possible for engines to disagree on whether a move is good or bad, as they do in the above example and others?

> to create better chess engine like what do even talking about here????

The ability to create better chess engines necessarily implies that chess engines can and do make mistakes, contrary to what you asserted.

> are you saying just because there are older bad engine that mean this thing is pointless ????

No. What I'm saying is that your explanation for why chess engines are better than humans is wrong. Chess engines are not better than humans because they have solved chess in the mathematical sense; chess engines are better than humans because they search the state space faster and more efficiently than humans (at least until you reach 7 pieces on the board).

> up until that point that you know high level chess is brute force games and therefore solvable math

"Solvable" and "solved" are two very different things. Chess is solvable, in theory. Chess is very far from being solved.

[0]: https://www.chess.com/computer-chess-championship#event=309&...

[1]: https://www.chess.com/computer-chess-championship#event=309&...

Re: Cloudflare outage on November 18, 2025 post mortem

#453

Post by agentifysh »

kylegalbraith wrote: post #448

The outage sucked for everyone. The root cause also feels like something they could have caught much earlier in a canary rollout from my reading of this. All that said, to have an outage reported turned around practically the same day, that is this detailed, is quite impressive. Here's to hoping they make their changes from this learning, and we don't see this exact failure mode again.

how would you build redundancy around cloudflare failing?

i think this is happening way too frequently

meanwhile VPS, dedicated servers hum along without any issues

i dont want to use kubernetes but if we have to build mission critical systems doesn't seem like building on cloudflare is going to cut it

Re: Cloudflare outage on November 18, 2025 post mortem

#454

Post by dehrmann »

never_inline wrote: post #377
bigstrat2003 wrote: post #315

Earlier quoted context omitted.

I'm with you! Checked exceptions are actually good and the hate for them is super short sighted. The exact same criticisms levied at checked exceptions apply to static typing in general, but people acknowledge the great value static types have for preventing errors at compile time. Checked exceptions have that same value, but are dunked on for some reason.

The dislike is probably because of 2 reasons. 1. in most cases they don't want to handle `InterruptedException` or `IOException` and yet need to bubble them up. In that case the code is very verbose. 2. it makes lambdas and functions incompatible. So eg: if you're passing a function to forEach, you're forced to wrap it in runtime exception. 3. Due to (1) and (2), most people become lazy and do `throws Exception` whic…

> 2. it makes lambdas and functions incompatible.

This is true, but the hate predated lambdas in Java.

Re: Cloudflare outage on November 18, 2025 post mortem

#455

Post by ChrisMarshallNY »

monocularvision wrote: post #358
ChrisMarshallNY wrote: post #190

Earlier quoted context omitted.

Swift has implicit unwrap (!), and explicit unwrap (?). I don't like to use implicit unwrap. Even things that are guaranteed to be there, I treat as explicit (For example, (self.view?.isEnabled ?? false) , in a view controller, instead of self.view.isEnabled ). I always redefine @IBOutlets from: @IBOutlet weak var someView! to: @IBOutlet weak var someView? I'm kind of a "belt & suspenders" type of guy.

So what happens if it ends up being nil? How does your app react? In this particular case, I would rather crash. It’s easier to spot in a crash report and you get a nice stack trace. Silent failure is ultimately terrible for users. Note: for the things I control I try to very explicitly model state in such a way as I never need to force unwrap at all. But for things beyond my control like this situation, I would rath…

> Silent failure is ultimately terrible for users.

Agreed.

Unfortunately, crashes in iOS are “silent failures,” and are a loss of control.

What this practice does, is give me the option to handle the failure “noisily,” and in a controlled manner; even if just emitting a log entry, before calling a system failure. That can be quite helpful, in threading. Also, it gives me the option to have a valid value applied, if there’s a structural failure.

But the main reason that I do that with @IBOutlets, is that it forces me to acknowledge, throughout the rest of the code, that it’s an optional. I could always treat implicit optionals as if they were explicit, anyway. This just forces me to.

I have a bunch of practices that folks can laugh at, but my stuff works pretty effectively, and I sleep well.

Re: Cloudflare outage on November 18, 2025 post mortem

#456

Post by dehrmann »

gwbas1c wrote: post #281
dehrmann wrote: post #231

Earlier quoted context omitted.

It's the same blind spot people have to Java's checked exceptions. People commonly resort to Pokemon exception handling and either blindly ignoring or rethrowing as a runtime exception. When Rust got popular, I was a bit confused by people talking about how great Result it's essentially a checked exception without a stack trace.

It's a lot lighter: a stack trace takes a lot of overhead to generate; a result has no overhead for a failure. The overhead (panic) only comes once the failure can't be handled. (Most books on Java/C# don't explain that throwing exceptions has high performance overhead.) Exceptions force a panic on all errors, which is why they're supposed to be used in "exceptional" situations. To avoid exceptions when an error is e…

> a stack trace takes a lot of overhead to generate

Can't Hotspot not generate the stack trace when it knows the exception will be caught and the stack trace ignored?

Re: Cloudflare outage on November 18, 2025 post mortem

#457

Post by guiriduro »

bobbylarrybobby wrote: post #345
AgentME wrote: post #214

Earlier quoted context omitted.

This is assuming that the process could have done anything sensible while it had the malformed feature file. It might be in this case that this was one configuration file of several and maybe the program could have been built to run with some defaults when it finds this specific configuration invalid, but in the general case, if a program expects a configuration file and can't do anything without it, panicking is a n…

I don't know too much about how the feature file distribution works but in the event of failure to read a new file, wouldn't logging the failure and sticking with the previous version of the file be preferable?

That's exactly the point (ie just prior to distribution) where a simple sanity check should have been run and the config replacement/update pipeline stopped on failure. When they introduced the 200 entry limit memory optimised feature loader it should have been a no-brainer to insert that sanity check in the config production pipeline.

Re: Cloudflare outage on November 18, 2025 post mortem

#458

Post by aaronmdjones »

philipwhiuk wrote: post #274
eastdakota wrote: post #35

Earlier quoted context omitted.

Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.

Why was Warp in London disabled temporarily. No mention of that change was discussed in the RCA despite it being called out in an update. For London customers this made the impact more severe temporarily.

Probably because it was the London team that was actively investigating the incident and initially came to the conclusion that it may be a DDoS while being unable to authenticate to their own systems.

Re: Cloudflare outage on November 18, 2025 post mortem

#459

Post by ozgrakkurt »

vsgherzi wrote: post #22

Why does cloudflare allow unwraps in their code? I would've assumed they'd have clippy lints stopping that sort of thing. Why not just match with { ok(value) => {}, Err(error) => {} } the function already has a Result type. At the bare minimum they could've used an expect("this should never happen, if it does database schema is incorrect"). The whole point of errors as values is preventing this kind of thing.... It w…

And the error magically disappears when the function returns it?

Re: Cloudflare outage on November 18, 2025 post mortem

#460

Post by uecker »

jryio wrote: post #426
samdoesnothing wrote: post #375

Earlier quoted context omitted.

> Pause for a moment and think about what a C++ implementation of a globally distributed network ingress proxy service would look like - and how many memory vulnerabilities there would be… I shudder at the thought I mean thats an unfalsifiable statement, not really fair. C is used to successfully launch spaceships. Whereas we have a real Rust bug that crashed a good portion of the internet for a significant amount of…

Only formal proof languages are immune to such properties. Therefore all languages are poorly designed by your metric. Consider that the set of possible failures enabled by language design should be as small as possible. Rust's set is small enough while also being productive. Until another breakthrough in language design as impactful as the borrow checker is invented, I don't imagine more programmers will be able to…

I would say the impact of the borrow checker is exaggerated.
Post reply on HN