Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

841–850 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#841

Post by jamincan »

Rohansi wrote: post #188
Dylan16807 wrote: post #172

Earlier quoted context omitted.

Some call points to a function that returns a Result will never return an Error. Some call points to a function that returns an int will never return -2. Sometimes you know things the type system does not know.

The difference is functions which return Result have explicitly chosen to return a Result because they can fail. Sure, it might not fail in the current implementation and/or configuration, but that could change later and you might not know until it causes problems. The type system is there to help you - why ignore it?

As a hypothetical example, when making a regex, I call `Regex::new(r"/d+")` which returns a result because my regex could be malformed and it could miscompile. It is entirely reasonable to unwrap this, though, as I will find out pretty quickly that it works or fails once I test the program.

Re: Cloudflare outage on November 18, 2025 post mortem

#842

Post by [deleted] »

tptacek wrote: post #825
JB_Dev wrote: post #447

Earlier quoted context omitted.

Code and Config should be treated similarly. If you would use a ring based rollout, canaries, etc for safely changing your code, then any config that can have the same impact must also use safe rollout techniques.

You're the nth person on this thread to say that and it doesn't make sense. Events that happen multiple times per second change data that you would call "configuration" in systems like these. This isn't `sendmail.cf`. If you want to say that systems that light up hundreds of customers, or propagate new reactive bot rules, or notify a routing system that a service has gone down are intrinsically too complicated, that'…

[deleted]

Re: Cloudflare outage on November 18, 2025 post mortem

#843

Post by ignoramous »

eastdakota wrote: post #684
SerCe wrote: post #9

As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.

* published less than 12 hours from when the incident began. Proud of the team for pulling together everything so quickly and clearly.

That's all well & good, but I'm curious...

> Spent some time after we got things under control talking to customers. Then went home.

What did sama / Fidji say? ;) Turnstile couldn't have been worth that.

Re: Cloudflare outage on November 18, 2025 post mortem

#844

Post by noname120 »

eastdakota wrote: post #689
ynx0 wrote: post #314

Earlier quoted context omitted.

How do you guys handle redaction? I'm sure even when trusted individuals are in charge of authoring, there's still a potential of accidental leakage which would probably be best mitigated by a team specifically looking for any slip ups. Thanks for the insight.

Team has a good sense, typically. In this case, the names of the columns in the Bot Management feature table seemed sensitive. The person who included that in the master document we were working from added a comment: “Should redact column names.” John and I usually catch anything the rest of the team may have missed. For me, pays to have gone to law school, but also pays to have studied Computer Science in college an…

Could you elaborate a bit on how going to law school helped? Was it because it made it easier for you to communicate and align with your CLO?

Re: Cloudflare outage on November 18, 2025 post mortem

#845

Post by frumplestlatz »

quotemstr wrote: post #711
frumplestlatz wrote: post #549

Earlier quoted context omitted.

I have to disagree that unwrap is ever OK. If you have to use unwrap, your types do not match your problem. Fix them. You have encoded invariants in your types that do not match reality. Change your API boundary, surface the discrepancy between your requirements and the potential failing case at the edges where it can be handled. If you need the value, you need to handle the case that it’s not available explicitly. Y…

> If you have to use unwrap, your types do not match your problem The problem starts with Rust stdlib. It panics on allocation failure. You expect Rust programmers to look at stdlib and not imitate it? Sure, you can try to taboo unwrap(), but 1) it won't work, and 2) it'll contort program design in places where failure really is a logic bug, not a runtime failure, and for which unwrap() is actually appropriate. The r…

> 1) it won't work

Of course it will. I've built enormous systems, including an entire compiler, without once relying on the local language equivalent of `.unwrap()`.

> 2) it'll contort program design in places where failure really is a logic bug, not a runtime failure, and for which unwrap() is actually appropriate.

That's a failure to model invariants in your API correctly.

> ... have them ship a language that makes error propagation the default and syntactically marks infallible cleanup paths --- like C++ with noexcept.

Unchecked exceptions aren't a solution. They're a way to avoid taking the thought, time, and effort to model failure paths, and instead leave that inherent unaddressed complexity until a runtime failure surprises users. Like just happened to Cloudflare.

Re: Cloudflare outage on November 18, 2025 post mortem

#846

Post by NetMageSCW »

markhandoff wrote: post #617

Dear Matthew Prince, don't you think we (the ones affected by your staff's mistake) should get some sort of compensation??? Yours truly, a Cloudflare client who lost money during the November 18th outage.

What does your contract with Cloudflare say?

Re: Cloudflare outage on November 18, 2025 post mortem

#847

Post by jamincan »

hu3 wrote: post #707
chippiewill wrote: post #540

Earlier quoted context omitted.

No, the "if it compiles, it works" is genuinely about the program being correct rather than just free of memory errors, but it's more of a hyperbolic statement than a statement of fact. It's a common thing I've experienced and seen a lot of others say that the stricter the language is in what it accepts the more likely it is to be correct by the time you get it to run. It's not just a Rust thing (although I think Rus…

Everyone understands Rust doesn't offer such guarantees. Even more now after this outage. But it's a fact that "if it compiles it runs" is often associated with Rust, in HN at least. A quick Algolia search tells me that.

I have definitely noticed this when I've tried doing Advent of Code in Rust - by the time my code compiles it typically send out the right answer. It doesn't help me once I don't know whatever algorithm I need to reach for in order to solve it before the heat death of the universe, but it is a somewhat magical feeling when it lasts.

Re: Cloudflare outage on November 18, 2025 post mortem

#848

Post by NetMageSCW »

oskarkk wrote: post #438
keiywuvfwofw wrote: post #399

This post was written by chatgpt?? https://blog.cloudflare.com/18-november-2025-outage/#:~:text...

Here's a random post from their blog by the same author from 2017 with an em dash: > As we wrote before, we believe Blackbird Tech's dangerous new model of patent trolling — where they buy patents and then act their own attorneys in cases — may be a violation of the rules of professional ethics. https://blog.cloudflare.com/patent-troll-battle-update-doubl... ChatGPT didn't invent the em dash, some people were always…

Too many think AI instead of horses.

Re: Cloudflare outage on November 18, 2025 post mortem

#850

Post by flerchin »

Based on this writeup it seems that Cloudflare defaults to a 0 score for bot prevention if there's a failure. Could instead it default to a passing score? Default open instead of default closed? This would have been a non-event to a lot of websites if that change was made.
Post reply on HN