Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

861–870 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#861

Post by JuniperMesos »

StopDisinfo910 wrote: post #596
JuniperMesos wrote: post #533

Earlier quoted context omitted.

Let's try to make effect systems happen quicker than that. > I mean this Cloudfare outage probably cost millions of dollars of damage in aggregate between lost revenue and lost productivity. How much of that will they actually have to pay? Probably nothing, because most paying customers of cloudflare are probably signing away their rights to sue Cloudflare for damages by being down for a while when they purchase Clou…

> I honestly have a hard time suggesting that those customers are individually wrong to do so - Cloudflare isn't down that often, and whatever amount it cost any individual customer by being down today might be more than offset by the DDOS protection they're buying. We have collectively agreed to a world where software service providers have no incentive to be reliable as they are shielded from the consequences of th…

> I want software provider to be liable for the damage they cause and minimum quality regulation on par with an actual engineering discipline. I have always been astounded that nearly all software licences start with extremely broad limitation of liability provisions and people somehow feel fine with it. Try to extend that to any other product you regularly use in your life and see how that makes you fell.

So do you want to make it illegal to punish GNU GPL licensed software because that license has a warranty disclaimer? Do you want to make it illegal for a company like Cloudflare to use open source licensed software with similar warranty disclaimers, or for the SLA agreements and penalties for violating them that they make with their own paying customers to be legally unenforceable? What if I just have a personal website and I break the javascript on it because I was careless, how should that be legally treated?

I'm not against research into more reliable software or using better engineering techniques that result in more reliable software. What I'm concerned about is the regulatory regime - in other words, what software it is or is not legal to write or sell for money - and how to properly incentivize software service providers to use techniques that result in more reliable software without causing a bunch of bad second order effects.

Re: Cloudflare outage on November 18, 2025 post mortem

#862

Post by twoodfin »

abalone wrote: post #859
twoodfin wrote: post #686

Earlier quoted context omitted.

But the rapid deployment mechanism for bot features wasn’t where the bug was introduced. In fact, the root bug (faulty assumption?) was in one or more SQL catalog queries that were presumably written some time ago. (Interestingly the analysis doesn’t go into how these erroneous queries made it into production OR whether the assumption was “to spec” and it’s the security principal change work that was faulty. Seems mo…

It was a change to the database that is used to generate a bot management config file. That file was the proximate cause for the panics. The kind of observability that would have helped here is “panics are elevated and here are the binary and config changes that preceded it,” along with a rollback runbook for it all. Generally I would say we as an industry are more nonchalant about config changes vs binary changes. W…

I am genuinely curious (albeit skeptical!) how anyone like Cloudflare could make that kind of feedback loop work at scale.

Even only in CF’s “critical path” there must be dozens of interconnected services and systems. How do you close the loop between an observed panic at the edge and a database configuration change N systems upstream?

Re: Cloudflare outage on November 18, 2025 post mortem

#863

Post by littlestymaar »

wrs wrote: post #834
littlestymaar wrote: post #648

Earlier quoted context omitted.

I don't understand your point: panic! is akin to expect : you think about it consciously, use it explicitly and you write down a panic message explaining its rational. unwrap isn't like that.

It should be. If you aren’t treating it exactly the same as panic and expect, that’s what I’m calling the “blind spot”. And why should you have to make up a message every time when the backtrace is going to tell you what was wrong?

> And why should you have to make up a message every time when the backtrace is going to tell you what was wrong?

The message isn't really here to be displayed during a crash (since the crash should never happen in the first place), it's here to communicate the invariant in the code, to the developer reading and modifying it later on.

Re: Cloudflare outage on November 18, 2025 post mortem

#864

Post by JuniperMesos »

otabdeveloper4 wrote: post #705
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

Oh come on, stop spreading FUD. Rust programs are 100% immune to crashes and bugs, they have memory safety (c). Also, exception handling is hard and lame. We don't need exceptions, just add a "match" block after every line in your program.

What's the point of this sarcastic comment? Do you think that some people claim that Rust's memory safety guarantees mean that a Rust program is incapable of crashing or having a bug? This is a dumb thing to claim certainly, but I'm not aware of anyone actually making this claim.

I'm also not sure what you're getting at with the comment about exception handling being lame. I think the ML/Haskell inspired model that Rust uses of having a parameterized Result type for fallible operations is generally better than exceptions for a variety of reasons (although maybe better Exception semantics could help with some of this), but what does this have to do with match blocks?

Re: Cloudflare outage on November 18, 2025 post mortem

#865

Post by literalAardvark »

benjiro wrote: post #723
sofixa wrote: post #595

Earlier quoted context omitted.

> There are many self-hosted alternatives to protect against botnet Whatever you do, unless you have their bandwidth capacity, at some point those "self-hosted" will get flooded with traffic.

As yourself more the question, is your service that important to need 99.999% uptime? Because i get the impression that people are so fixated on this uptime concept, that the idea of being down for a few hours is the most horrible issue in the world. To the point that they rather hand over control of their own system to a 3th party, then accept a downtime. The fact that cloudflare can literally ready every bit of com…

There are many systems that benefit from ddos protection without actually needing the high uptime.

The bandwidth costs of a ddos alone would close down a small shop.

Cloudflare provide an incredibly good service with a great track record, and sometimes shit happens.

Re: Cloudflare outage on November 18, 2025 post mortem

#867

Post by loglog »

vbezhenar wrote: post #585
shadowmatter wrote: post #404

Earlier quoted context omitted.

Even in lowly Java, they later added to Optional the orElseThrow() method since the name of the get() method did not connote the impact of unwrapping an empty Optional.

I've found both methods very useful. I'm using `get()` when I've checked that the value is present and I don't expect any exceptions. I'm using `orElseThrow()` when I actually expect that value can be absent and throwing is fine. Something like if (userOpt.isPresent()) { var user = userOpt.get(); var accountOpt = accountRepository.selectAccountOpt(user.getId()); var account = accountOpt.orElseThrow(); } Idea checks i…

While the `Optional` API is generally pretty inconvenient (compared e.g. to Kotlin), it does offer the more precise `ifPresent`.
Post reply on HN