Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

71–80 of 265 posts

Re: Cloudflare outage should not have happened

#71

Earlier quoted context omitted.

Ok, let's start off with holding them to the same standards as avionics software development. The formal verification can wait.

Agreed. I left out any commentary on `.unwrap()` from my original comment, but it’s an obvious example of something that should never have appeared in critical code.

Rust needs to get rid of .unwrap() and its kin. They're from pre-1.0 Rust, before many of the type system features and error handling syntax sugar were added.

There's no reason to use them as the language provides lots of safer alternatives. If you do want to trigger a panic, you can, but I'd also ask - why?

Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other than malloc failures. Any function that then calls a panicky function needs to be similarly marked. In doing this, we can statically be certain no such methods are called if we want to be rid of the behavior.

Perhaps something like:

    panic fn my_panicky_function() {
      None.unwrap(); // NB: `unwrap()` is also marked `panic` in stdlib 
    }

    fn my_safe_function() {
      // with a certain compiler or Crates flag, this would fail to compile
      // as my_safe_function isn't annotated as `panic`
      my_panicky_function() 
    }
The ideal future would be to have code that is 100% panic free.

Re: Cloudflare outage should not have happened

#72
post #9

* The unwrap() in production code should have never passed code review. Damn, it should have been flagged by a linter. * The deployment should have followed the blue/green pattern, limiting the blast radius of a bad change to a subset of nodes. * In general, a company so much at the foundational level of internet connectivity should not follow the "move fast, break things" pattern. They did not have an overwhelming r…

> the blue/green pattern

?

Re: Cloudflare outage should not have happened

#73

Earlier quoted context omitted.

Ok, let's start off with holding them to the same standards as avionics software development. The formal verification can wait.

Agreed. I left out any commentary on `.unwrap()` from my original comment, but it’s an obvious example of something that should never have appeared in critical code.

And it's so easy to avoid, as well.

    #![deny(clippy::unwrap_used)]
or

    cargo clippy -- -D clippy::unwrap_used
Put that in your CI pipeline, and voila. Global crash averted.

Re: Cloudflare outage should not have happened

#74
post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

This bug might not have, but others would. Formal verification methods still rely on humans to input the formal specification, which is where problems happen.

As others point out, if they didn't really ship fast, they certainly would not have become profitable, and they would definitely not have captured the market to the extent they have.

But really, if the market was more distributed, and Cloudflare commanded 5% of the web as the biggest player, any single outage would have been limited in impact. So it's also about market behaviour: yet "nobody is fired for choosing IBM" as it used to go 40 years ago.

Re: Cloudflare outage should not have happened

#76
post #61

Earlier quoted context omitted.

Are Cloudflare's customers willing to pay avionics software level prices?

Given that Cloudflare's market cap is 1/2 of Boeing's and they are not making a physical product I would say: Clearly, yes.

The vast majority of Cloudflare's "customers" are paying 0 to 20 dollars a month, for virtually the same protection coverage and features as most of their 200 dollars/mo customers. That's not remotely in the realm of avionics price structure, be it software or hardware.

Re: Cloudflare outage should not have happened

#77

Earlier quoted context omitted.

Precisely. This is key infrastructure we're talking about not some kind of webshop.

Yeah but the anti-DDOS feature needs to react to new methods all the time, it's not a static thing you build once and it works forever. An insulin pump is very different. Your human body, insulin, and physics aren't changing any time soon.

You are simplifying the control software of an insulin point to a degree that does not match reality. I'm saying that because I actually reviewed the code of one and the amount of safety consciousness on display there was off the charts compared to what you usually encounter in typical web development. You also under-estimate the dynamic nature of the environment these pumps operate in as well as the amount of contingency planning that they embody, failure modes of each and every part in the pump were taken into consideration, and there are more such parts that you are most likely aware of. This includes material defects, defects as a result from abuse, wear & tear, parts being simply out of spec and so on.

To see this as the typical firmware that ships with say a calculator or a watch is to diminish the accomplishment considerably.

Re: Cloudflare outage should not have happened

#79
post #72
post #9

* The unwrap() in production code should have never passed code review. Damn, it should have been flagged by a linter. * The deployment should have followed the blue/green pattern, limiting the blast radius of a bad change to a subset of nodes. * In general, a company so much at the foundational level of internet connectivity should not follow the "move fast, break things" pattern. They did not have an overwhelming r…

> the blue/green pattern ?

This specific terminology was new to me, too: https://en.wikipedia.org/wiki/Blue%E2%80%93green_deployment

Re: Cloudflare outage should not have happened

#80
post #71

Earlier quoted context omitted.

Agreed. I left out any commentary on `.unwrap()` from my original comment, but it’s an obvious example of something that should never have appeared in critical code.

Rust needs to get rid of .unwrap() and its kin. They're from pre-1.0 Rust, before many of the type system features and error handling syntax sugar were added. There's no reason to use them as the language provides lots of safer alternatives. If you do want to trigger a panic, you can, but I'd also ask - why? Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other…

I'd say the equivalent of Erlang's supervisor trees is what is needed but once you go that route you might as well use Erlang.
Post reply on HN