Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

81–90 of 265 posts

Re: Cloudflare outage should not have happened

#81
post #46

Earlier quoted context omitted.

Like your comment? j/k :) I'm using this incident to draw attention to Rust's panic behavior. Rust could use additional language features to help us write mostly panic-free* code and statically catch even transitive dependencies that might subject us to unnecessary panics. We've been talking about it on our team and to other Rust folks, and I think it's worth building a proposal around. Rust should have a way to stat…

It's already in the box... there's a bunch of options from unwrap_or, etc... to actually checking the error result and dealing with it cleanly... that's not what happened. Not to mention the possibility of just bumping up through Result chaining with an app specific error model. The author chose neither... likely because they want the app to crash/reload from an external service. This is often the best approach to an…

> This is often the best approach to an indeterminate or unusable state/configuration.

The engineers had more semantic tools at their disposal for this than a bare `unwrap()`.

This was a systems failure. A better set of tools in Rust would have helped mitigate some of the blow.

`unwrap()` is from pre-1.0 Rust, before many of the type system-enabled error safety features existed. And certainly before many of the idiomatic syntactic sugars were put into place.

I posted in another thread that Rust should grow annotation features to allow us to statically rid or minimize our codebase of panic behavior. Outside of malloc failures, we should be able to constrain or rid large classes of them with something like this:

    panic fn my_panicky_function() {
      None.unwrap(); // NB: `unwrap()` is also marked `panic` in stdlib 
    }

    fn my_safe_function() {
      // with a certain compiler or Crates flag, this would fail to compile
      // as my_safe_function isn't annotated as `panic`
      my_panicky_function() 
    }
Obviously just an idea, but something like this would be nice. We should be able to do more than just linting, and we should have tools that guarantee transitive dependencies can't blow off our feet with panic shotguns.

In any case, until something is done, this is not the last time we'll hear unwrap() horror stories.

Re: Cloudflare outage should not have happened

#82
post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

I agree with you.

I would just add that I've noticed organizations tend to calcify as they get bigger and older. Kind of like trees, they start out as flexible saplings, and over time develop hard trunks and branches. The rigidity gives them stability.

You're right that there's no way they could have gotten to where they are if they had prioritized data integrity and formal verification in all their practices. Now that they have so much market share, they might collapse under their own weight if their trunk isn't solid. Maybe investing in data integrity and strongly typed, functional programming that's formally verifiable is what will help them keep their market share.

Cultures are hard to change and I'm not suggesting an expectation for them to change beyond what is feasible or practical. I don't lead an engineering organization like it so I'm definitely armchairing here. I just see some of the logic of the argument that them adopting some of these methods would probably benefit everyone using their services.

Re: Cloudflare outage should not have happened

#83
I think the author is trying to apply a preconceived cause on to the cloudflare outage, but there’s not a fit.

E.g., they should try to work through how their own suggested fix would actually ensure the problem couldn’t happen. I don’t believe it would… lack of nullable fields and normalization typically simplify relational logic, but hardly prevent logical errors. Formal verification can prove your code satisfies a certain formal specification, but doesn’t prove your specification solves your business problem (or makes sense at all, in fact).

Re: Cloudflare outage should not have happened

#84
post #17
post #3

I agree it should not have happened, but I don’t agree that the database schema is the core problem. The “logical single point of failure” here was created by the rapid, global deployment process. If you don’t want to take down all of prod, you can’t update all of prod at the same time. Gradual deployments are a more reliable defense against bugs than careful programming.

>Gradual deployments are a more reliable defense against bugs than careful programming The challenge, as I understand it, is that the feature in question had an explicit requirement of fast, wide deployment because of the need to react in real time to changing external attacker behaviors.

Yeah, I don’t know how fast “fast” needs to be in this system; but my understanding is this particular failure would have been seen immediately on the first replica. The progression could still be aggressive after verifying the first wave.

Re: Cloudflare outage should not have happened

#85
post #9

* The unwrap() in production code should have never passed code review. Damn, it should have been flagged by a linter. * The deployment should have followed the blue/green pattern, limiting the blast radius of a bad change to a subset of nodes. * In general, a company so much at the foundational level of internet connectivity should not follow the "move fast, break things" pattern. They did not have an overwhelming r…

Unless you work at Cloudflare it seems very unlikely that you have enough information about systems and tradeoffs there to make these flat assertions about what "should have" happened. Systems can do worse things than crashing in response to unexpected states. Blue/green deployment isn't always possible (eg due to constrained compute resources) or practical (perhaps requiring greatly increased complexity), and is by no means the only approach to reducing deploy risk. We don't know that any of the related code was shipped with a "move fast, break things" mindset; the most careful developers still write bugs.

Actually learning from incidents and making systems more reliable requires curiosity and a willingness to start with questions rather than mechanically applying patterns. This is standard systems-safety stuff. The sort of false confidence involved in making prescriptions from afar suggests a mindset I don't want anywhere near the operation of anything critical.

Re: Cloudflare outage should not have happened

#86
post #6

Earlier quoted context omitted.

I disagree. I learnt good stuff from this article and it’s enough.

> I disagree. I learnt good stuff from this article and it’s enough. That's perfectly fine. It's also besides the point though. You can learn without reading random people online cynically shit talking others as a self promotion strategy. This is junior dev energy manifesting junior level understanding of the whole problem domain. There's not a lot to learn from claims that boil down to "don't have bugs".

I laughed out loud when he said Cloudflare should have formally verified its systems.

Re: Cloudflare outage should not have happened

#87
post #7

"This massive, accomplished engineering team whose software operates at a scale nearly no one else operates at missed this basic thing" is a hell of a take.

Honestly it's a quite lukewarm take.

See for example https://danluu.com/algorithms-interviews/. This sort of thing happens constantly.

Re: Cloudflare outage should not have happened

#88
post #81

Earlier quoted context omitted.

It's already in the box... there's a bunch of options from unwrap_or, etc... to actually checking the error result and dealing with it cleanly... that's not what happened. Not to mention the possibility of just bumping up through Result chaining with an app specific error model. The author chose neither... likely because they want the app to crash/reload from an external service. This is often the best approach to an…

> This is often the best approach to an indeterminate or unusable state/configuration. The engineers had more semantic tools at their disposal for this than a bare `unwrap()`. This was a systems failure. A better set of tools in Rust would have helped mitigate some of the blow. `unwrap()` is from pre-1.0 Rust, before many of the type system-enabled error safety features existed. And certainly before many of the idiom…

What you're suggesting is perfectly reasonable, I wouldn't object to labeling methods that can panic via bare unwrap...

I'm just saying that having a program immediately exit (via panic or not) could very well be the appropriate behavior.

Re: Cloudflare outage should not have happened

#89
post #76

Earlier quoted context omitted.

Given that Cloudflare's market cap is 1/2 of Boeing's and they are not making a physical product I would say: Clearly, yes.

The vast majority of Cloudflare's "customers" are paying 0 to 20 dollars a month, for virtually the same protection coverage and features as most of their 200 dollars/mo customers. That's not remotely in the realm of avionics price structure, be it software or hardware.

It is the aggregate they pay that counts here, not the individual payments.

A better comparison would be to compare this to airline passengers paying for their tickets, they pay a few hundred bucks in the expectation that they will arrive at their destination.

Besides, it is not the customers that determine Cloudflare's business model, Cloudflare does. Note that their whole business is to prevent outages and that as soon as they become the cause of an outage they have invalidated their whole reason for existence. Of course you could then turn this into a statistical argument that as long as they prevent more outages than they cause that they are a net benefit but that's not what this discussion is about, it is first and foremost about the standard of development they are held up against.

Ericsson identified similar issues in their offering long ago and created a very capable solution and I'm wondering if that would not have been a better choice for this kind of project, even if it would have resulted in more resource consumption.

Post reply on HN