Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

611–620 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#611

Post by eertami »

tonyhart7 wrote: post #430
aw1621107 wrote: post #385

Earlier quoted context omitted.

> because these thing called BEST MOVE and BAD MOVE there in chess The thing is that there is no known general objective criteria for "best" and "bad" moves. The best we have so far is based on engine evaluations, but as I said before that is because chess engines are better at searching the board's state space than humans, not because chess engines have solved chess in the mathematical sense. Engines are quite capab…

"The thing is that there is no known general objective criteria for "best" and "bad" moves." are you playing chess or not?????? if you playing chess then its oblivious how to differentiate bad move and best move Yes it is objective, these thing called best move not without reason "If you think chess engines are infalliable, then why does the Top Chess Engine Championship exist?" to create better chess engine like wha…

> if you playing chess up to a decent level 1700+ (like me), you know that these argument its wrong and I assure you to learn chess to a decent level

In a fascinating coincidence, there is a tonyhart7 on both chess.com and lichess, and they have been banned for cheating on both websites.

Re: Cloudflare outage on November 18, 2025 post mortem

#612

Post by arkanovicz »

Interesting technical insight, but I would be curious to hear firsthand accounts from the teams on the ground, particularly regarding how the engineers felt the increasing pressure, frantically refreshing their dashboards, searching for phantom DDoS, scrolling codes updates...

Re: Cloudflare outage on November 18, 2025 post mortem

#613

Post by jve »

Carriethebest wrote: post #587
jve wrote: post #582

Earlier quoted context omitted.

> I'm migrating my customers off Cloudflare. Is that an overreaction? Name me global, redundant systems that have not (yet) failed. And if you used cloudflare to protect against botnet and now go off cloudflare... you are vulnerable and may experience more downtime if you cannot swallow the traffic. I mean no service have 100% uptime - just that some have more nines than others.

There are many self-hosted alternatives to protect against botnet. We don't have to use cloudflare. Everthing is under their control!

Well if you self host DDoS protection service, that would be VERY expensive. You would need rent rack space along with a very fast internet connection at multiple data centers to host this service.

Re: Cloudflare outage on November 18, 2025 post mortem

#614

Post by [deleted] »

hnarn wrote: post #508

> That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network. > The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That…

[deleted]

Re: Cloudflare outage on November 18, 2025 post mortem

#615

Post by huijzer »

polack wrote: post #570
abalone wrote: post #380

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…

> They failed on so many levels here.

That's often the case with human error as especially aviation safety experts know: https://en.wikipedia.org/wiki/Swiss_cheese_model

Re: Cloudflare outage on November 18, 2025 post mortem

#616

Post by l___l »

> The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.

What could have prevented this failure?

Cloudflare's software could have included a check that refused to generate the feature file if it's size was higher than the limit.

A testcase could have caught this.

Re: Cloudflare outage on November 18, 2025 post mortem

#618

Post by Ciantic »

twhitmore wrote: post #589
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

Interesting to see Rust error handling flunk out in practice. It may be that forcing handling at every call tends to makes code verbose, and devs insensitized to bad practice. And the diagnostic Rust provided seems pretty garbage. There is bad practice here too -- config failure manifesting as request failure, lack of failing to safe, unsafe rollout, lack of observability. Back to language design & error handling. My…

> It may be that forcing handling at every call tends to makes code verbose

Rust has a lot of helpers to make it less verbose, even that error they demonstrate could've been written in some form `...code()?` with `?` helper that would have propagated the error forwards.

However I do acknowledge that writing Error types is boring sometimes so people don't bother to change their error types and just unwrap. But even my dinghy little apps for my personal use I do simple serach `unwrap` and make sure I have as few as possible.

Re: Cloudflare outage on November 18, 2025 post mortem

#619

Post by tuetuopay »

andy_ppp wrote: post #603
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

This is why the Erlang/Elixir methodology of having supervision and letting things crash gracefully is so useful. You can either handle every single error gracefully or handle crashing gracefully - it's much easier and more realistic in large codebases to do the later.

This would not have helped: the code would crash before doing anything useful at all.

If anything, the "crash early" mentality may even be nefarious: instead of handling the error and keeping the old config, you would spin on trying to load a broken config on startup.

Re: Cloudflare outage on November 18, 2025 post mortem

#620

Post by PhilipRoman »

throwaway2037 wrote: post #461
echelon wrote: post #425

Earlier quoted context omitted.

See the immediately preceding sentence. I'm fine with allocation failures. I don't want stupid unwrap()s, improper slice access, or other stupid and totally preventable behavior. There are things inside the engineer's control. I want that to not panic.

Your pair of posts is very interesting to me. Can you share with me: What is your programming environment such that you are "fine with allocation failures"? I'm not doubting you, but for me, if I am doing systems programming with C or C++, my program is doomed if a malloc fails! When I saw your post, I immediately thought: Am I doing it wrong? If I get a NULL back from malloc(), I just terminate with an error message…

Not to mention overcommit has become standard behavior on many systems, so you wouldn't even get a NULL unless you really tried.
Post reply on HN