Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

191–200 of 265 posts

Re: Cloudflare outage should not have happened

#191

Adding distinct or group by to a query is not some advanced technic comments are suggesting. It does not slow down development one bit, if you expect distinct result you put explicit distinct in the query, it's not a "safety measure for insulin pumps". Scratching my head what I've missed here, please enlighten me.

DISTINCT would just be masking the query bug

Random DISTINCT is usually a code smell that indicates an incorrect join / filter

Re: Cloudflare outage should not have happened

#192
post #182

Earlier quoted context omitted.

You said: > Rust needs to get rid of .unwrap() and its kin. Now you say: > You can keep unwrap() and panics. So which is it? > I just want a static first class method to ensure it never winds up in our code or in the dependencies we consume. Now this is absolutely a reasonable request. But it's not an easy one to provide depending on how you go about it. For example, I'd expect your suggestion in your other comment t…

> unwrap() was only a symptom of an already bad state causing an error that the service couldn't recover from. This would have been as much of an unrecoverable error if it was reported in any other way. The mechanisms needed to either prevent it or recover are much more nuanced than just whether it's an unwrap or Result. This sounds like the kind of failure Bobby Tables warned about a long time ago. An entire new, sa…

In a local sense, "quit out safely when the config is corrupt" is pretty correct.

Coordinated systems that test and rollback are way beyond the scope of what a compiler can check.

Re: Cloudflare outage should not have happened

#193
https://blog.cloudflare.com/18-november-2025-outage/

"Customers deployed on the new FL2 proxy engine, observed HTTP 5xx errors. Customers on our old proxy engine, known as FL, did not see errors, but bot scores were not generated correctly, resulting in all traffic receiving a bot score of zero."

This simply means, the exception handling quality of your new FL2 is non-existent and is not at par / code logic wise similar to FL.

I hope it was not because of AI driven efficiency gains.

Re: Cloudflare outage should not have happened

#194

Earlier quoted context omitted.

It would not because it would be a compile time error rather than run time error which is a completely different beast if I understand the argument correctly.

What would be a compile time error? The compiler rejecting unwrap? And then you fix that by bubbling the error case up, which fixes the compiler error and leaves you with a runtime error again. But one that's less ergonomic. You can't force a config file loaded at run time to be correct at compile time. You can only decide what you're going to do about the failure.

The point they are - trying, apparently - making is that if you had a flag or an annotation that you could make to a function that you do not want that function to be built on top of anything that can 'unwrap' that you can rule out some of these cases of unexpected side effects.

Re: Cloudflare outage should not have happened

#195
post #43
post #9

* The unwrap() in production code should have never passed code review. Damn, it should have been flagged by a linter. * The deployment should have followed the blue/green pattern, limiting the blast radius of a bad change to a subset of nodes. * In general, a company so much at the foundational level of internet connectivity should not follow the "move fast, break things" pattern. They did not have an overwhelming r…

unwrap() and the family of methods like it are a Rust anti-pattern from the early days of Rust. It dates back to before many of the modern error-handling and safety-conscious features of the language and type system. Rust is being pulled in so many different directions from new users that the language perhaps never originally intended. Some engineers will be fine with panicky behavior, but a lot of others want to be…

This is starting to sound a lot like checked exceptions in Java.

Re: Cloudflare outage should not have happened

#196
post #182

Earlier quoted context omitted.

You said: > Rust needs to get rid of .unwrap() and its kin. Now you say: > You can keep unwrap() and panics. So which is it? > I just want a static first class method to ensure it never winds up in our code or in the dependencies we consume. Now this is absolutely a reasonable request. But it's not an easy one to provide depending on how you go about it. For example, I'd expect your suggestion in your other comment t…

> unwrap() was only a symptom of an already bad state causing an error that the service couldn't recover from. This would have been as much of an unrecoverable error if it was reported in any other way. The mechanisms needed to either prevent it or recover are much more nuanced than just whether it's an unwrap or Result. This sounds like the kind of failure Bobby Tables warned about a long time ago. An entire new, sa…

The fact that this wasn't RCE or anything other than denial of service is a raging success of Rust.

“If it compiles it’s probably correct” has always been a tongue-in-cheek pithy exaggeration. I heard it among Haskell programmers long before I heard it in the context of Rust. And guess what? Haskell programs have bugs too.

Re: Cloudflare outage should not have happened

#197
post #182

Earlier quoted context omitted.

> unwrap() was only a symptom of an already bad state causing an error that the service couldn't recover from. This would have been as much of an unrecoverable error if it was reported in any other way. The mechanisms needed to either prevent it or recover are much more nuanced than just whether it's an unwrap or Result. This sounds like the kind of failure Bobby Tables warned about a long time ago. An entire new, sa…

In a local sense, "quit out safely when the config is corrupt" is pretty correct. Coordinated systems that test and rollback are way beyond the scope of what a compiler can check.

What about “detect when the content isn’t correct and take protective measures so that a core service of the global internet _doesn’t_ crash?” Wasn’t that the whole point of rust? I’ll repeat again “if it compiles it is almost absolutely correct” is a mantra I see on hn daily.

Apparently that isn’t true.

Edit: isn’t the whole idea of C/C++ being flawed pivoted around memory management and how flawed the languages are? Wasn’t the whole point of rust to eliminate that whole class of errors? XSS and buffer overflows are almost always caused by “malformed” outside input. Rust apparently doesn’t protect against that.

Re: Cloudflare outage should not have happened

#198
post #171

Earlier quoted context omitted.

Rust generally forces you to acknowledge these faults. The problem is managing them in a sane way, which for Rust in many cases simply is failing loudly. Compared to than many other languages which preferring chugging along and hoping that no downstream corruption happens.

[flagged]

There were several different components internal to Cloudflare that failed in a complex distributed systems context; the Rust failure is garnering more attention partially because it was a very legible failure, which also makes it easy for Cloudflare to fix this bug and all similar bugs quickly. The Cloudflare postmortem is a pro-Rust argument. It's also an argument that too many institutions rely on Cloudflare, which is a harder problem to solve.

Re: Cloudflare outage should not have happened

#199

Earlier quoted context omitted.

Yeah but the anti-DDOS feature needs to react to new methods all the time, it's not a static thing you build once and it works forever. An insulin pump is very different. Your human body, insulin, and physics aren't changing any time soon.

You are simplifying the control software of an insulin point to a degree that does not match reality. I'm saying that because I actually reviewed the code of one and the amount of safety consciousness on display there was off the charts compared to what you usually encounter in typical web development. You also under-estimate the dynamic nature of the environment these pumps operate in as well as the amount of contin…

If humans beings had a small chance to transform into say quadrupeds or suddenly grow tenatacles, extra hearts, organs, etc., in any given year… then wouldn’t designing a safe insulin pump literally be impossible?

Re: Cloudflare outage should not have happened

#200
post #189

Earlier quoted context omitted.

[flagged]

But it wasn't the culprit, the code could have been in anything, or could have bubbled up errors to main, and it still would have failed with for an incorrect config.

Right. So the language that espoused to eliminate errors that took down large positions of the internet, failed.

The specifics matter of course, but the mantra of rust as some safe language that should never have allowed something like this to happen, happened.

I vote we rename rust to “rustantic” in honor of human hubris.

Post reply on HN