Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

Post reply on HN
Print view View original

131–140 of 265 posts

Re: Cloudflare outage should not have happened

#131

Post by burntsushi »

echelon wrote: post #71
frumplestlatz wrote: post #47

Earlier quoted context omitted.

Agreed. I left out any commentary on `.unwrap()` from my original comment, but it’s an obvious example of something that should never have appeared in critical code.

Rust needs to get rid of .unwrap() and its kin. They're from pre-1.0 Rust, before many of the type system features and error handling syntax sugar were added. There's no reason to use them as the language provides lots of safer alternatives. If you do want to trigger a panic, you can, but I'd also ask - why? Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other…

I'm on libs-api. We will never get rid of unwrap(). It is absolutely okay to use unwrap(). It's just an assertion. Assertions appear in critical code all the time, including the standard library. Just like it's okay to use `slice[i]`.

Re: Cloudflare outage should not have happened

#132

Post by burntsushi »

lenkite wrote: post #126
frumplestlatz wrote: post #47

Earlier quoted context omitted.

Agreed. I left out any commentary on `.unwrap()` from my original comment, but it’s an obvious example of something that should never have appeared in critical code.

ON HN, just a couple of years ago, a famous Rust programmer said that it is OK to use unwrap. Rustaceans supported this position. Cloudflare merely followed the community standard. Using unwrap() in Rust is Okay https://news.ycombinator.com/item?id=32385102 https://burntsushi.net/unwrap/

I already had a conversation with the GP specifically: https://news.ycombinator.com/item?id=45979127

They aren't presenting a coherent philosophy. And when asked for examples, or to engage directly with examples in my blog, they can't or won't do it.

But yes, of course it's okay to use unwrap(). It's just an assertion. Assertions are fine.

Re: Cloudflare outage should not have happened

#133

Post by svieira »

echelon wrote: post #71
frumplestlatz wrote: post #47

Earlier quoted context omitted.

Agreed. I left out any commentary on `.unwrap()` from my original comment, but it’s an obvious example of something that should never have appeared in critical code.

Rust needs to get rid of .unwrap() and its kin. They're from pre-1.0 Rust, before many of the type system features and error handling syntax sugar were added. There's no reason to use them as the language provides lots of safer alternatives. If you do want to trigger a panic, you can, but I'd also ask - why? Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other…

All that means is that the `Failure` bubbles up to the very top of `main` (in this scenario) because we're only caring about the happy path (because we can't conceive of what the unhappy path should be other than "crash") and then hits the `panic("Well, that's unexpected")` explicitly in Place B rather than Place A (the `.unwrap`). I'm not sure how that's _better_.

Re: Cloudflare outage should not have happened

#134

Post by frumplestlatz »

dpark wrote: post #63
frumplestlatz wrote: post #32

Earlier quoted context omitted.

I work in formal verification at a FAANG. It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths, and even then, the properties asserted by formal verification are often quite a bit less powerful than necessary to truly guarantee something useful. I want formal verification everywhere. I believe in provable correctness. I…

> we know how to do it better, but nearly all of even the smartest engineers we can hire are not smart enough. This seems like a contradiction. If the smartest engineers you can hire are not smart enough to work within formal verification constraints then we in fact do not know how to do this. If formal verification hinges on having perfect engineers then it’s useless because perfect engineers wouldn’t need formal ve…

> If formal verification hinges on having perfect engineers then it’s useless because perfect engineers wouldn’t need formal verification.

It doesn’t hinge on having perfect engineers.

It hinges on engineers being able to model problems algebraically and completely, prove the equivalence of multiple such models at different layers of abstraction (including equivalence to the original code), and then prove that useful properties hold across those models.

Re: Cloudflare outage should not have happened

#135

Post by echelon »

burntsushi wrote: post #131
echelon wrote: post #71

Earlier quoted context omitted.

Rust needs to get rid of .unwrap() and its kin. They're from pre-1.0 Rust, before many of the type system features and error handling syntax sugar were added. There's no reason to use them as the language provides lots of safer alternatives. If you do want to trigger a panic, you can, but I'd also ask - why? Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other…

I'm on libs-api. We will never get rid of unwrap(). It is absolutely okay to use unwrap(). It's just an assertion. Assertions appear in critical code all the time, including the standard library. Just like it's okay to use `slice[i]`.

This is the Hundred Billion Dollar unwrap() Bug.

You can keep unwrap() and panics. I just want a static first class method to ensure it never winds up in our code or in the dependencies we consume.

I have personally been involved in nearly a billion dollars of outages myself and am telling you there are simple things the language can do to help users purge their code of this.

This is a Rust foot gun.

A simple annotation and compiler flag to disallow would suffice. It needs to handle both my code and my dependencies. We can build it ourselves as a hack, but it will never be 100% correct.

This is why I want it:

https://news.ycombinator.com/item?id=46060907

Re: Cloudflare outage should not have happened

#136

Post by frumplestlatz »

burntsushi wrote: post #132
lenkite wrote: post #126

Earlier quoted context omitted.

ON HN, just a couple of years ago, a famous Rust programmer said that it is OK to use unwrap. Rustaceans supported this position. Cloudflare merely followed the community standard. Using unwrap() in Rust is Okay https://news.ycombinator.com/item?id=32385102 https://burntsushi.net/unwrap/

I already had a conversation with the GP specifically: https://news.ycombinator.com/item?id=45979127 They aren't presenting a coherent philosophy. And when asked for examples, or to engage directly with examples in my blog, they can't or won't do it. But yes, of course it's okay to use unwrap(). It's just an assertion. Assertions are fine.

Result declares a type-level invariant — an assertion enforced by the compiler, not runtime — that the operation can fail.

Ignoring that is bypassing the type system. It means your types are either wrong, or your type system is incapable of modeling your true invariants.

In the case of the cloudflare error, their types were wrong. That was an avoidable failure. They needed to fix their type-level invariants, not yolo the issue with `.unwrap()`.

Your willful persistent lack of understanding doesn’t mean my philosophy is incoherent. Using `.unwrap()` is always an example of a failure to accurately model your invariants in the type system.

Re: Cloudflare outage should not have happened

#137

Post by burntsushi »

echelon wrote: post #135
burntsushi wrote: post #131

Earlier quoted context omitted.

I'm on libs-api. We will never get rid of unwrap(). It is absolutely okay to use unwrap(). It's just an assertion. Assertions appear in critical code all the time, including the standard library. Just like it's okay to use `slice[i]`.

This is the Hundred Billion Dollar unwrap() Bug . You can keep unwrap() and panics. I just want a static first class method to ensure it never winds up in our code or in the dependencies we consume. I have personally been involved in nearly a billion dollars of outages myself and am telling you there are simple things the language can do to help users purge their code of this. This is a Rust foot gun. A simple annota…

You said:

> Rust needs to get rid of .unwrap() and its kin.

Now you say:

> You can keep unwrap() and panics.

So which is it?

> I just want a static first class method to ensure it never winds up in our code or in the dependencies we consume.

Now this is absolutely a reasonable request. But it's not an easy one to provide depending on how you go about it. For example, I'd expect your suggestion in your other comment to be a non-starter because of the impact it will have on language complexity. But that doesn't mean there isn't a better way. (I just don't know what it is.)

This is a classic motte and bailey. You come out with a bombastic claim like "remove unwrap and its ilk," but when confronted, you retreat to the far more reasonable, "I just want tools to detect and prevent panicking branches." If you had said the latter, I wouldn't have even responded to you. I wouldn't have even batted an eye.

> This is the Hundred Billion Dollar unwrap() Bug.

The Cloudflare bug wasn't even caused by unwrap(). unwrap() is just its manifestation. From a Cloudflare employee:

> In this case the unwrap() was only a symptom of an already bad state causing an error that the service couldn't recover from. This would have been as much of an unrecoverable error if it was reported in any other way. The mechanisms needed to either prevent it or recover are much more nuanced than just whether it's an unwrap or Result.

Re: Cloudflare outage should not have happened

#138

Post by dpark »

echelon wrote: post #121
dpark wrote: post #120

Earlier quoted context omitted.

So slicing is forbidden in this scheme? But not malloc? This doesn’t seem to be a principled stance on making the language safer. It feels a bit whack-a-mole. “Unwrap is pretty easy to give up. I could live without slicing. Malloc seems hard though. I don’t want to give that up.”

I posted about why this is important for distributed systems engineering: https://news.ycombinator.com/item?id=46060907 Malloc is fine. We can and do monitor that. It's these undetectable runtime logic problems that are land mines. In distributed systems, these can cause contagion and broad outages. Recovering can be very difficult and involve hours of complex steps across dozens of teams. Meanwhile you're losing mil…

I don’t actually work in Rust. I think I understand what you’re going for, though. The choice to use panic as a way of propagating errors is fundamentally problematic when it can arise from code you don’t control and potentially cannot even inspect.

I don’t necessarily agree that malloc should be okay (buggy code could try to allocate a TB of memory and OOMKiller won’t fix it) but I can understand that it’s probably workable in most cases.

Unfortunately I think the fix here would require a compatibility break.

Re: Cloudflare outage should not have happened

#139

Post by DrSusanCalvin »

engineeringwoke wrote: post #128
DrSusanCalvin wrote: post #123

Earlier quoted context omitted.

Care to elaborate? Perhaps the tools to do this in practice aren't there (which just shows how young the field of software "engineering" really is), but what consensus are you talking about and how is it an obstacle to verifying code? Most of the web follows standards and protocols, which actually sort of a prerequisite for communications across different systems...

Basically the modern web uses orchestration, for pretty much everything. Usually Kubernetes is doing that. Theoretically protocols like RAFT are formally verifiable, but their implementations in orchestration tools like etcd have not been, and I would go so far as to say that that is an impossible task. Therefore, the entire exercise is kind of silly.

Thanks, interesting. However, that just seems like a protocol like any other, with no real reason why you "can't" formally verify it. Is there something special about a consensus algorithm / protocol that makes it any more difficult to verify than any other algorithm which doesn't yet have a formally verified implementation?

Edit: https://link.springer.com/chapter/10.1007/978-3-319-48989-6_...

Re: Cloudflare outage should not have happened

#140

Post by burntsushi »

frumplestlatz wrote: post #136
burntsushi wrote: post #132

Earlier quoted context omitted.

I already had a conversation with the GP specifically: https://news.ycombinator.com/item?id=45979127 They aren't presenting a coherent philosophy. And when asked for examples, or to engage directly with examples in my blog, they can't or won't do it. But yes, of course it's okay to use unwrap(). It's just an assertion. Assertions are fine.

Result declares a type-level invariant — an assertion enforced by the compiler, not runtime — that the operation can fail. Ignoring that is bypassing the type system. It means your types are either wrong, or your type system is incapable of modeling your true invariants. In the case of the cloudflare error, their types were wrong. That was an avoidable failure. They needed to fix their type-level invariants, not yolo…

Your definition of "correct" is completely incoherent. Just because an invariant that could be modeled by a type system is not modeled by the type system in any given scenario does not make it incorrect.

You can't engage with my examples and you provide none of your own. So continuing discussion with you is a waste of time.

Post reply on HN