Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

Post reply on HN
Print view View original

211–220 of 265 posts

Re: Cloudflare outage should not have happened

#211

Post by igregoryca »

gishh wrote: post #200
saati wrote: post #189

Earlier quoted context omitted.

But it wasn't the culprit, the code could have been in anything, or could have bubbled up errors to main, and it still would have failed with for an incorrect config.

Right. So the language that espoused to eliminate errors that took down large positions of the internet, failed. The specifics matter of course, but the mantra of rust as some safe language that should never have allowed something like this to happen, happened. I vote we rename rust to “rustantic” in honor of human hubris.

The only languages that eliminate logic bugs are formally verified ones, as the article points out. (And even then, your program is only as correct as your specification.) Ordinary Rust code is not formally verified. Anyone who claims Rust eliminates errors is either very naive or lying.

Type-safe Rust code is free from certain classes of errors. But that goes out the window the moment you parse input from the outside, because Rust types can enforce invariants (i.e. internal consistency), but input has no invariants. Rust doesn't ban you from crashing the program if you see input that violates an invariant. I don't know of any mainstream language that forbids crashing the program. (Maybe something like Ada? Not sure.)

I don't understand why you bemoan that Rust hasn't solved this problem, because it seems nigh unsolvable.

Re: Cloudflare outage should not have happened

#212

Post by svieira »

echelon wrote: post #148
svieira wrote: post #133

Earlier quoted context omitted.

All that means is that the `Failure` bubbles up to the very top of `main` (in this scenario) because we're only caring about the happy path (because we can't conceive of what the unhappy path should be other than "crash") and then hits the `panic("Well, that's unexpected")` explicitly in Place B rather than Place A (the `.unwrap`). I'm not sure how that's _better_.

Not really. Handler and middleware can handle this without much ceremony. The user gets to, and is informed of and encouraged to, choose. We also don't get surprised at runtime. It's in the AST and we know at compile time. The right API signature helps the engineer think about things and puts them in the correct headspace for systems thinking. If something is panicking under the hood, the thought probably doesn't eve…

Yes, but my point is that without a reasonable supervision tree and crash boundary the difference between a composition of Result-returning functions that bottoms out in main's implicit panic and an explicit panic is nil operationally.

While lexically the unwrap actually puts the unhandledness of the error case as close to the source of the issue's source as possible. In order to get that lexical goodness you'd need something much more fine grained than Result.

Re: Cloudflare outage should not have happened

#213

Post by kjgkjhfkjf »

mosura wrote: post #55
vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

Software people, especially coming through Rust, are falling into the old trap of believing if code is bug free it is reliable: it isn’t because there is a world of faults outside, including but not limited to the developer intentions. This inverts everything because structuring to be fault tolerant, of the right things, changes what is a good idea almost entirely.

To be fair to Rust, the issue was an "unwrap" in the Rust code[0]. "unwrap" means "if the operation did not succeed then panic". Production Rust code should not use "unwrap", and should instead have logic to handle the failure case.

You don't need exotic formal verification methods to enforce this best practice. You just need a linter.

[0] https://blog.cloudflare.com/18-november-2025-outage/#memory-...

Re: Cloudflare outage should not have happened

#214

Post by gishh »

burntsushi wrote: post #196
gishh wrote: post #182

Earlier quoted context omitted.

> unwrap() was only a symptom of an already bad state causing an error that the service couldn't recover from. This would have been as much of an unrecoverable error if it was reported in any other way. The mechanisms needed to either prevent it or recover are much more nuanced than just whether it's an unwrap or Result. This sounds like the kind of failure Bobby Tables warned about a long time ago. An entire new, sa…

The fact that this wasn't RCE or anything other than denial of service is a raging success of Rust. “If it compiles it’s probably correct” has always been a tongue-in-cheek pithy exaggeration. I heard it among Haskell programmers long before I heard it in the context of Rust. And guess what? Haskell programs have bugs too.

> “If it compiles it’s probably correct” has always been a tongue-in-cheek pithy exaggeration.

If you say so, I believe you. That isn’t how it comes across in daily, granted pithy, discourse around here.

I have a lot of respect for you Andrew, not meaning to attack you per se. You surely can see the irony in the internet falling over because of an app written in rust, and all that comes with this whole story, no?

Re: Cloudflare outage should not have happened

#215

Post by burntsushi »

gishh wrote: post #214
burntsushi wrote: post #196

Earlier quoted context omitted.

The fact that this wasn't RCE or anything other than denial of service is a raging success of Rust. “If it compiles it’s probably correct” has always been a tongue-in-cheek pithy exaggeration. I heard it among Haskell programmers long before I heard it in the context of Rust. And guess what? Haskell programs have bugs too.

> “If it compiles it’s probably correct” has always been a tongue-in-cheek pithy exaggeration. If you say so, I believe you. That isn’t how it comes across in daily, granted pithy, discourse around here. I have a lot of respect for you Andrew, not meaning to attack you per se. You surely can see the irony in the internet falling over because of an app written in rust, and all that comes with this whole story, no?

Nope. Because you've completely mischaracterized not only the actual problem here, but the value proposition of Rust. You're tilting at windmills.

Nobody credible has ever said that Rust will fix all your problems 100% of the time. If that's what you inferred was being sold based on random HN commentary, then you probably want to revisit how you absorb information.

Rust has always been about reducing bugs, with a specific focus on bugs as a result of undefined behavior. It has never, isn't and will never be able to eliminate all bugs. At minimum, you need formal methods for that.

Rust programs can and will have bugs as a result of undefined behavior. The value proposition is that their incidence should be markedly lower than programs written in C or C++ (i.e., implementations of languages that are memory unsafe by default).

Re: Cloudflare outage should not have happened

#216

Post by syockit »

jacquesm wrote: post #77
simlevesque wrote: post #38

Earlier quoted context omitted.

Yeah but the anti-DDOS feature needs to react to new methods all the time, it's not a static thing you build once and it works forever. An insulin pump is very different. Your human body, insulin, and physics aren't changing any time soon.

You are simplifying the control software of an insulin point to a degree that does not match reality. I'm saying that because I actually reviewed the code of one and the amount of safety consciousness on display there was off the charts compared to what you usually encounter in typical web development. You also under-estimate the dynamic nature of the environment these pumps operate in as well as the amount of contin…

The point still stands. The human body still isn't going change. That's why insulin pump can afford to have all kinds of rigorous engineering, while web-facing infrastructure on the other hand needs to be able to quickly adapt to changes.

Re: Cloudflare outage should not have happened

#217

Post by zbentley »

frumplestlatz wrote: post #141
burntsushi wrote: post #140

Earlier quoted context omitted.

Your definition of "correct" is completely incoherent. Just because an invariant that could be modeled by a type system is not modeled by the type system in any given scenario does not make it incorrect. You can't engage with my examples and you provide none of your own. So continuing discussion with you is a waste of time.

Invariants aren’t invariant if they’re variant. This is literally what “invariant” means, and what a type system is built to model. Declaring an invariant in the type system that you then violate is not correct code. I truly can’t even begin to guess at why you’re so voracious in your defense of this particularly poor practice. [edit] HN rate limits kicking in, so here’s my reply. I work for a FAANG but I’m not going…

> I work for a FAANG but I’m not going to say which one. You or a relative are, with almost 100% certainty, relying on code written to that philosophy, by me, daily and widely.

Cool story bro.

Like, even interpreted maximally charitably, your statement still doesn’t provide GP’s requested published code. Not “take my word for it” ostensibly deployed software—code; the discussion here is about code constructs for modeling invariants, not solely about runtime behavior.

I’d be interested to see that code discussed in context of the blog post GP linked, which seems to make a compelling argument.

Re: Cloudflare outage should not have happened

#218

Post by zbentley »

12_throw_away wrote: post #106
echelon wrote: post #71

Earlier quoted context omitted.

Rust needs to get rid of .unwrap() and its kin. They're from pre-1.0 Rust, before many of the type system features and error handling syntax sugar were added. There's no reason to use them as the language provides lots of safer alternatives. If you do want to trigger a panic, you can, but I'd also ask - why? Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other…

> There's no reason to use [panics] as the language provides lots of safer alternatives. Dunno ... I think runtime assertions and the ability to crash a misbehaving program are a pretty important part of the toolset. If rust required `Result`s to be wired up up and down the entire call tree for the privilege of using a runtime assertion, I think it would be a lot less popular, and probably less safe in practice. > Al…

> a mechanism to prove that code can or cannot panic would be great

As appealing as the idea of a #[cfg(nopanic)] enforcement mechanism is, I think linting for panic() is the optimum, actually.

With a more rigidly enforced nopanic guarantee, I worry that some code and coders would start to rely on it (informally, accidentally, or out of ignorance) as a guarantee of completion, not return behavior. And that’s bad; adding language features which can easily be misconstrued to obscure the fact that all programs can terminate at any time is dangerous.

Lints, on the other hand, can be loud and enforced (and tools to recursively lint source-available dependencies exist), but few people mistake them for runtime behavior enforcement.

Re: Cloudflare outage should not have happened

#219

Post by zbentley »

jacquesm wrote: post #80
echelon wrote: post #71

Earlier quoted context omitted.

Rust needs to get rid of .unwrap() and its kin. They're from pre-1.0 Rust, before many of the type system features and error handling syntax sugar were added. There's no reason to use them as the language provides lots of safer alternatives. If you do want to trigger a panic, you can, but I'd also ask - why? Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other…

I'd say the equivalent of Erlang's supervisor trees is what is needed but once you go that route you might as well use Erlang.

I’m not sure that panic (speaking generally about the majority of its uses and the spirit of the law; obviously 100% of code does not obey this) is the equivalent of an Erlang process crash in most cases. Rather, I think unwrap()/panic are usually used in ways more similar to erlang:halt/1.

Re: Cloudflare outage should not have happened

#220

Post by gishh »

burntsushi wrote: post #215
gishh wrote: post #214

Earlier quoted context omitted.

> “If it compiles it’s probably correct” has always been a tongue-in-cheek pithy exaggeration. If you say so, I believe you. That isn’t how it comes across in daily, granted pithy, discourse around here. I have a lot of respect for you Andrew, not meaning to attack you per se. You surely can see the irony in the internet falling over because of an app written in rust, and all that comes with this whole story, no?

Nope. Because you've completely mischaracterized not only the actual problem here, but the value proposition of Rust. You're tilting at windmills. Nobody credible has ever said that Rust will fix all your problems 100% of the time. If that's what you inferred was being sold based on random HN commentary, then you probably want to revisit how you absorb information. Rust has always been about reducing bugs, with a spe…

> If that's what you inferred was being sold based on random HN commentary, then you probably want to revisit how you absorb information.

Heard, chef.

Post reply on HN