Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

Post reply on HN
Print view View original

141–150 of 265 posts

Re: Cloudflare outage should not have happened

#141

Post by frumplestlatz »

burntsushi wrote: post #140
frumplestlatz wrote: post #136

Earlier quoted context omitted.

Result declares a type-level invariant — an assertion enforced by the compiler, not runtime — that the operation can fail. Ignoring that is bypassing the type system. It means your types are either wrong, or your type system is incapable of modeling your true invariants. In the case of the cloudflare error, their types were wrong. That was an avoidable failure. They needed to fix their type-level invariants, not yolo…

Your definition of "correct" is completely incoherent. Just because an invariant that could be modeled by a type system is not modeled by the type system in any given scenario does not make it incorrect. You can't engage with my examples and you provide none of your own. So continuing discussion with you is a waste of time.

Invariants aren’t invariant if they’re variant.

This is literally what “invariant” means, and what a type system is built to model.

Declaring an invariant in the type system that you then violate is not correct code. I truly can’t even begin to guess at why you’re so voracious in your defense of this particularly poor practice.

[edit]

HN rate limits kicking in, so here’s my reply. I work for a FAANG but I’m not going to say which one. You or a relative are, with almost 100% certainty, relying on code written to that philosophy, by me, daily and widely.

Re: Cloudflare outage should not have happened

#142

Post by spwa4 »

nine_k wrote: post #9

* The unwrap() in production code should have never passed code review. Damn, it should have been flagged by a linter. * The deployment should have followed the blue/green pattern, limiting the blast radius of a bad change to a subset of nodes. * In general, a company so much at the foundational level of internet connectivity should not follow the "move fast, break things" pattern. They did not have an overwhelming r…

* The step in front of this query created updates to policies. It should have been limited in the number of changes it would do at once (and ideally per hour and per day and so on), and if it goes over that limit, stop updating, alert and wait until explicitly unblocked. DO NOT generate invalid config and start using that invalid config, use the previous one that worked and alert.

If this happens during startup use a default one.

That would still create impact (customers and developers would not see updates propagate), but would avoid destroying the service. When it comes to outages, people need to learn to go over what happens in the case of violating an invariant and look at what gets sacrificed in those cases, to make sure the answer isn't "the whole service".

If I get to be impolite, you do this because software architects, as seems to be the case here, often choose "crash and destroy the service" when their invariants are violated instead of "stop doing shit and alert" when faced with an unknown problem, or a problem they can't deal with.

This also requires test-crashing. You introduce an assert? Great! The more the merrier, seriously, you should have lots of them. BUT you will be including a test that the world doesn't end when your assert is hit.

Re: Cloudflare outage should not have happened

#143

Post by burntsushi »

frumplestlatz wrote: post #141
burntsushi wrote: post #140

Earlier quoted context omitted.

Your definition of "correct" is completely incoherent. Just because an invariant that could be modeled by a type system is not modeled by the type system in any given scenario does not make it incorrect. You can't engage with my examples and you provide none of your own. So continuing discussion with you is a waste of time.

Invariants aren’t invariant if they’re variant. This is literally what “invariant” means, and what a type system is built to model. Declaring an invariant in the type system that you then violate is not correct code. I truly can’t even begin to guess at why you’re so voracious in your defense of this particularly poor practice. [edit] HN rate limits kicking in, so here’s my reply. I work for a FAANG but I’m not going…

Show me code you've published that is used by real people in real systems that follows the philosophy you've espoused here. Otherwise I'm calling shenanigans.

Re: Cloudflare outage should not have happened

#144

Post by dpark »

frumplestlatz wrote: post #134
dpark wrote: post #63

Earlier quoted context omitted.

> we know how to do it better, but nearly all of even the smartest engineers we can hire are not smart enough. This seems like a contradiction. If the smartest engineers you can hire are not smart enough to work within formal verification constraints then we in fact do not know how to do this. If formal verification hinges on having perfect engineers then it’s useless because perfect engineers wouldn’t need formal ve…

> If formal verification hinges on having perfect engineers then it’s useless because perfect engineers wouldn’t need formal verification. It doesn’t hinge on having perfect engineers. It hinges on engineers being able to model problems algebraically and completely, prove the equivalence of multiple such models at different layers of abstraction (including equivalence to the original code), and then prove that useful…

If the smartest engineers cannot do it, it doesn’t work.

This isn’t even getting to the practical question of whether it’s worth doing, given the significant additional cost. If the smartest folks you can find are not smart enough to use the framework then it’s useless.

Maybe this means the tooling is insufficient. Maybe it means the field isn’t mature enough. Whatever, if you need an IQ two standard deviations above normal and 10x as long it’s not real world useable today.

Re: Cloudflare outage should not have happened

#145

Post by jacquesm »

dpark wrote: post #57
jacquesm wrote: post #37

Earlier quoted context omitted.

Ok, let's start off with holding them to the same standards as avionics software development. The formal verification can wait.

I don’t understand why anyone should want this. Why should normal engineering efforts be held to the same standards as life-critical systems? Why would anyone expect that CloudFlare DDoS protection be built to the standards of avionics equipment? Also if we’re being fair, avionics software is far narrower in scope than just “software in general”. And even with that Boeing managed to kill a bunch of people with shitty…

> I don’t understand why anyone should want this.

That's ok, but then you should bow out of the conversation, which is between people that do understand why anyone should want this.

To have predictable behavior is a must have in some industries, less in others. At the level of infrastructure that is deemed critical by some - and I'm curious what JGC's position on this is - the ability to avoid this kind of outage carries a lot of value. The fact that you do not see that CF has achieved life-critical reach is one that tells me that most of this effort is probably going to waste, but I trust that John does see it my way and realizes that if there are ways to avoid these kind of issues they should be researched. Because service uptime is something very important to companies like Cloudflare.

Boeing managed to kill a bunch of people with shitty business practices, not with shitty software, the software did what it was built to do. It is the whole process around that software as well as the type certification process and regulatory oversight that failed dramatically.

Re: Cloudflare outage should not have happened

#146

Post by jacquesm »

frumplestlatz wrote: post #32
jacquesm wrote: post #25

Earlier quoted context omitted.

When you're powering this large a fraction of the internet is it even an option not to work like that? You'd think that with that kind of market cap resource constraints should no longer be holding you back from doing things properly.

I work in formal verification at a FAANG. It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths, and even then, the properties asserted by formal verification are often quite a bit less powerful than necessary to truly guarantee something useful. I want formal verification everywhere. I believe in provable correctness. I…

The big trick is - as far as I understand it - to acknowledge that systems fail and to engineer for dealing with those failures.

I support your efforts downthread for at least knowing whether or not underlying abstractions are able to generate a panic (which is a massive side effect) or are only able to return valid results or error flags. The higher level the abstraction the bigger the chance that there is a module somewhere in the stack that is able to blow it all up, at the highest level you can pretty much take it as read that this is the case.

So unless you engineer the whole thing from the ground up without any library modules it is impossible to guarantee that this is not the case and as far as I understand your argument you at least want to be informed when that is the case, or, alternatively, to cause the compiler to flag the situation down from your code as incompatible with the guarantees that you are asking for, is that a correct reading?

Re: Cloudflare outage should not have happened

#147

Post by jacquesm »

svieira wrote: post #133
echelon wrote: post #71

Earlier quoted context omitted.

Rust needs to get rid of .unwrap() and its kin. They're from pre-1.0 Rust, before many of the type system features and error handling syntax sugar were added. There's no reason to use them as the language provides lots of safer alternatives. If you do want to trigger a panic, you can, but I'd also ask - why? Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other…

All that means is that the `Failure` bubbles up to the very top of `main` (in this scenario) because we're only caring about the happy path (because we can't conceive of what the unhappy path should be other than "crash") and then hits the `panic("Well, that's unexpected")` explicitly in Place B rather than Place A (the `.unwrap`). I'm not sure how that's _better_.

It would not because it would be a compile time error rather than run time error which is a completely different beast if I understand the argument correctly.

Re: Cloudflare outage should not have happened

#148

Post by echelon »

svieira wrote: post #133
echelon wrote: post #71

Earlier quoted context omitted.

Rust needs to get rid of .unwrap() and its kin. They're from pre-1.0 Rust, before many of the type system features and error handling syntax sugar were added. There's no reason to use them as the language provides lots of safer alternatives. If you do want to trigger a panic, you can, but I'd also ask - why? Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other…

All that means is that the `Failure` bubbles up to the very top of `main` (in this scenario) because we're only caring about the happy path (because we can't conceive of what the unhappy path should be other than "crash") and then hits the `panic("Well, that's unexpected")` explicitly in Place B rather than Place A (the `.unwrap`). I'm not sure how that's _better_.

Not really. Handler and middleware can handle this without much ceremony. The user gets to, and is informed of and encouraged to, choose.

We also don't get surprised at runtime. It's in the AST and we know at compile time.

The right API signature helps the engineer think about things and puts them in the correct headspace for systems thinking. If something is panicking under the hood, the thought probably doesn't even occur to them.

Re: Cloudflare outage should not have happened

#149

Post by engineeringwoke »

DrSusanCalvin wrote: post #139
engineeringwoke wrote: post #128

Earlier quoted context omitted.

Basically the modern web uses orchestration, for pretty much everything. Usually Kubernetes is doing that. Theoretically protocols like RAFT are formally verifiable, but their implementations in orchestration tools like etcd have not been, and I would go so far as to say that that is an impossible task. Therefore, the entire exercise is kind of silly.

Thanks, interesting. However, that just seems like a protocol like any other, with no real reason why you "can't" formally verify it. Is there something special about a consensus algorithm / protocol that makes it any more difficult to verify than any other algorithm which doesn't yet have a formally verified implementation? Edit: https://link.springer.com/chapter/10.1007/978-3-319-48989-6_...

That would be like saying that you can verify the software that CERN uses to measure the Higgs Boson because we verified general relativity.

Re: Cloudflare outage should not have happened

#150

Post by locknitpicker »

DrSusanCalvin wrote: post #97
locknitpicker wrote: post #14

Earlier quoted context omitted.

> I disagree. I learnt good stuff from this article and it’s enough. That's perfectly fine. It's also besides the point though. You can learn without reading random people online cynically shit talking others as a self promotion strategy. This is junior dev energy manifesting junior level understanding of the whole problem domain. There's not a lot to learn from claims that boil down to "don't have bugs".

This article actually explains how this bug in particular could have been avoided. Sure you may not consider his approach realistic, but it's not at all saying "don't have bugs". In fact, not having formal verification or similar tooling in place, would be more like saying "just don't write buggy code".

> This article actually explains how this bug in particular could have been avoided.

Not really. The article is a textbook example of hindsight bias. It's a simplistic analysis of a far more complex problem that goes over the blogger's head, and results in a string of simplistic assertions that fail to address any of the issues. Read up on the definition of monday morning quarterback.

Post reply on HN