Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

Post reply on HN
Print view View original

181–190 of 265 posts

Re: Cloudflare outage should not have happened

#181

Post by burntsushi »

echelon wrote: post #43
nine_k wrote: post #9

* The unwrap() in production code should have never passed code review. Damn, it should have been flagged by a linter. * The deployment should have followed the blue/green pattern, limiting the blast radius of a bad change to a subset of nodes. * In general, a company so much at the foundational level of internet connectivity should not follow the "move fast, break things" pattern. They did not have an overwhelming r…

unwrap() and the family of methods like it are a Rust anti-pattern from the early days of Rust. It dates back to before many of the modern error-handling and safety-conscious features of the language and type system. Rust is being pulled in so many different directions from new users that the language perhaps never originally intended. Some engineers will be fine with panicky behavior, but a lot of others want to be…

> As it stands, I'm pretty mortified that some transitive dependency might use unwrap() deep in its internals.

You'll have to go without std and even the `core` library then.

Re: Cloudflare outage should not have happened

#182

Post by gishh »

burntsushi wrote: post #137
echelon wrote: post #135

Earlier quoted context omitted.

This is the Hundred Billion Dollar unwrap() Bug . You can keep unwrap() and panics. I just want a static first class method to ensure it never winds up in our code or in the dependencies we consume. I have personally been involved in nearly a billion dollars of outages myself and am telling you there are simple things the language can do to help users purge their code of this. This is a Rust foot gun. A simple annota…

You said: > Rust needs to get rid of .unwrap() and its kin. Now you say: > You can keep unwrap() and panics. So which is it? > I just want a static first class method to ensure it never winds up in our code or in the dependencies we consume. Now this is absolutely a reasonable request. But it's not an easy one to provide depending on how you go about it. For example, I'd expect your suggestion in your other comment t…

> unwrap() was only a symptom of an already bad state causing an error that the service couldn't recover from. This would have been as much of an unrecoverable error if it was reported in any other way. The mechanisms needed to either prevent it or recover are much more nuanced than just whether it's an unwrap or Result.

This sounds like the kind of failure Bobby Tables warned about a long time ago. An entire new, safe language was developed to prevent these kinds of failures. “If it compiles it’s probably correct” seems to be the mantra of rust. Nuts.

Re: Cloudflare outage should not have happened

#183

Post by mikestorrent »

jacquesm wrote: post #77
simlevesque wrote: post #38

Earlier quoted context omitted.

Yeah but the anti-DDOS feature needs to react to new methods all the time, it's not a static thing you build once and it works forever. An insulin pump is very different. Your human body, insulin, and physics aren't changing any time soon.

You are simplifying the control software of an insulin point to a degree that does not match reality. I'm saying that because I actually reviewed the code of one and the amount of safety consciousness on display there was off the charts compared to what you usually encounter in typical web development. You also under-estimate the dynamic nature of the environment these pumps operate in as well as the amount of contin…

Thanks for spelling that out. It's so often tempting to be reductionist about things, but there is often a tremendous amount of thankless engineering inside products that we are privileged to consider as being somewhat boring. It takes a lot of work to make something so dynamic and life-critical and make it reliable enough to be considered simple, when it's anything but.

Re: Cloudflare outage should not have happened

#184

Post by Dylan16807 »

jacquesm wrote: post #147
svieira wrote: post #133

Earlier quoted context omitted.

All that means is that the `Failure` bubbles up to the very top of `main` (in this scenario) because we're only caring about the happy path (because we can't conceive of what the unhappy path should be other than "crash") and then hits the `panic("Well, that's unexpected")` explicitly in Place B rather than Place A (the `.unwrap`). I'm not sure how that's _better_.

It would not because it would be a compile time error rather than run time error which is a completely different beast if I understand the argument correctly.

What would be a compile time error? The compiler rejecting unwrap? And then you fix that by bubbling the error case up, which fixes the compiler error and leaves you with a runtime error again. But one that's less ergonomic.

You can't force a config file loaded at run time to be correct at compile time. You can only decide what you're going to do about the failure.

Re: Cloudflare outage should not have happened

#185

Post by fwip »

echelon wrote: post #43
nine_k wrote: post #9

* The unwrap() in production code should have never passed code review. Damn, it should have been flagged by a linter. * The deployment should have followed the blue/green pattern, limiting the blast radius of a bad change to a subset of nodes. * In general, a company so much at the foundational level of internet connectivity should not follow the "move fast, break things" pattern. They did not have an overwhelming r…

unwrap() and the family of methods like it are a Rust anti-pattern from the early days of Rust. It dates back to before many of the modern error-handling and safety-conscious features of the language and type system. Rust is being pulled in so many different directions from new users that the language perhaps never originally intended. Some engineers will be fine with panicky behavior, but a lot of others want to be…

Does #[no_panic] do it for you? https://docs.rs/no-panic/latest/no_panic/

Re: Cloudflare outage should not have happened

#186

Post by saati »

echelon wrote: post #135
burntsushi wrote: post #131

Earlier quoted context omitted.

I'm on libs-api. We will never get rid of unwrap(). It is absolutely okay to use unwrap(). It's just an assertion. Assertions appear in critical code all the time, including the standard library. Just like it's okay to use `slice[i]`.

This is the Hundred Billion Dollar unwrap() Bug . You can keep unwrap() and panics. I just want a static first class method to ensure it never winds up in our code or in the dependencies we consume. I have personally been involved in nearly a billion dollars of outages myself and am telling you there are simple things the language can do to help users purge their code of this. This is a Rust foot gun. A simple annota…

Who lost a hundred billion dollars?

Re: Cloudflare outage should not have happened

#187

Post by mikestorrent »

saati wrote: post #180
gishh wrote: post #171

Earlier quoted context omitted.

[flagged]

Was it a memory error or a data race? No. Rust only promises that those won't happen in safe Rust. What is embarrassing is trying to pin this on a specific programming language.

[flagged]

Re: Cloudflare outage should not have happened

#188

Post by Dylan16807 »

echelon wrote: post #117
12_throw_away wrote: post #106

Earlier quoted context omitted.

> There's no reason to use [panics] as the language provides lots of safer alternatives. Dunno ... I think runtime assertions and the ability to crash a misbehaving program are a pretty important part of the toolset. If rust required `Result`s to be wired up up and down the entire call tree for the privilege of using a runtime assertion, I think it would be a lot less popular, and probably less safe in practice. > Al…

> I 100% agree that a mechanism to prove that code can or cannot panic would be great, but why would malloc be special here? Folks who are serious about preventing panics will generally use `no-std` in order to prevent malloc in the first place. In one of the domains I work in, a malloc failure and OOMkill are equivalent. We just restart the container. I've done all the memory pressure measurement ahead of time and r…

If you replace panic with a bespoke fallback or retry, have you really gained anything? You can still have all your services die at the same time, and you'll have even less of a smoking gun since you won't have a thousand stack traces pointing at the same line.

The core issue is that resilience to errors is hard, and you can't avoid that via choice of panic versus non-panic equivalents.

Re: Cloudflare outage should not have happened

#189

Post by saati »

mikestorrent wrote: post #187
saati wrote: post #180

Earlier quoted context omitted.

Was it a memory error or a data race? No. Rust only promises that those won't happen in safe Rust. What is embarrassing is trying to pin this on a specific programming language.

[flagged]

But it wasn't the culprit, the code could have been in anything, or could have bubbled up errors to main, and it still would have failed with for an incorrect config.

Re: Cloudflare outage should not have happened

#190

Post by kalkin »

nine_k wrote: post #152
kalkin wrote: post #85

Earlier quoted context omitted.

Unless you work at Cloudflare it seems very unlikely that you have enough information about systems and tradeoffs there to make these flat assertions about what "should have" happened. Systems can do worse things than crashing in response to unexpected states. Blue/green deployment isn't always possible (eg due to constrained compute resources) or practical (perhaps requiring greatly increased complexity), and is by…

Indeed, I never worked at Cloudflare. Still I have some nebulous idea about Cloudflare, and especially their scale. Systems can do worse things than crashing in response to unexpected states, but they can also do better to report them and terminate gracefully. Especially if the code runs on so many nodes, and the crash renders them unresponsive. Blue/green deployment isn't always possible, but my imagination is a bit…

I appreciate that this comment is much less prescriptive. I don't think I disagree with you about any general best practices here (although I do think unwrap can be fine when you can locally verify the error or nil case is unreachable but proving that to the compiler is impractical.)
Post reply on HN