Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

Post reply on HN
Print view View original

61–70 of 265 posts

Re: Cloudflare outage should not have happened

#61

Post by khuey »

jacquesm wrote: post #37
frumplestlatz wrote: post #32

Earlier quoted context omitted.

I work in formal verification at a FAANG. It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths, and even then, the properties asserted by formal verification are often quite a bit less powerful than necessary to truly guarantee something useful. I want formal verification everywhere. I believe in provable correctness. I…

Ok, let's start off with holding them to the same standards as avionics software development. The formal verification can wait.

Are Cloudflare's customers willing to pay avionics software level prices?

Re: Cloudflare outage should not have happened

#62

Post by pizlonator »

> No nullable fiels.

If you take away nullability, you eventually get something like a special state that denotes absence and either:

- Assertions that the absence never happens.

- Untested half-baked code paths that try (and fail) to handle absence.

> formally verified

Yeah, this does prevent most bugs.

But it's horrendously expensive. Probably more expensive than the occasional Cloudflare incident

Re: Cloudflare outage should not have happened

#63

Post by dpark »

frumplestlatz wrote: post #32
jacquesm wrote: post #25

Earlier quoted context omitted.

When you're powering this large a fraction of the internet is it even an option not to work like that? You'd think that with that kind of market cap resource constraints should no longer be holding you back from doing things properly.

I work in formal verification at a FAANG. It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths, and even then, the properties asserted by formal verification are often quite a bit less powerful than necessary to truly guarantee something useful. I want formal verification everywhere. I believe in provable correctness. I…

> we know how to do it better, but nearly all of even the smartest engineers we can hire are not smart enough.

This seems like a contradiction. If the smartest engineers you can hire are not smart enough to work within formal verification constraints then we in fact do not know how to do this.

If formal verification hinges on having perfect engineers then it’s useless because perfect engineers wouldn’t need formal verification.

Re: Cloudflare outage should not have happened

#65

Post by rdtsc »

ottah wrote: post #54
frumplestlatz wrote: post #32

Earlier quoted context omitted.

I work in formal verification at a FAANG. It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths, and even then, the properties asserted by formal verification are often quite a bit less powerful than necessary to truly guarantee something useful. I want formal verification everywhere. I believe in provable correctness. I…

I would argue the largest CDN provider in the world is a critical path.

I would guess at the individual team level they probably still behave like any other tech shop. When the end of the year comes the higher-ups still expect fancy features and accomplishments and saying "well, we spent months writing a page of TLA+ code" is not going to look as "flashy" as another team who delivered 20 new features. It would take someone from above to push and ask that other team who delivered 20 features, where is their TLA+ code verifying their correctness. But, how many people in the middle management chain would do that?

Re: Cloudflare outage should not have happened

#66

Post by jacquesm »

khuey wrote: post #61
jacquesm wrote: post #37

Earlier quoted context omitted.

Ok, let's start off with holding them to the same standards as avionics software development. The formal verification can wait.

Are Cloudflare's customers willing to pay avionics software level prices?

Given that Cloudflare's market cap is 1/2 of Boeing's and they are not making a physical product I would say: Clearly, yes.

Re: Cloudflare outage should not have happened

#67

Post by galleywest200 »

locknitpicker wrote: post #14
nmoura wrote: post #6

Earlier quoted context omitted.

I disagree. I learnt good stuff from this article and it’s enough.

> I disagree. I learnt good stuff from this article and it’s enough. That's perfectly fine. It's also besides the point though. You can learn without reading random people online cynically shit talking others as a self promotion strategy. This is junior dev energy manifesting junior level understanding of the whole problem domain. There's not a lot to learn from claims that boil down to "don't have bugs".

> You can learn without reading random people online

Somebody has to write something in the first place for one to learn from it, even if the writing is disagreeable.

Re: Cloudflare outage should not have happened

#68

Post by xoa »

SoftTalker wrote: post #15
vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

Why is being able to "capture the market" something we want to encourage? This leads to monopolies or oligopolies and makes possible various types of abuse that a free competitive market would normally correct. If you're going to step into the role of managing a large percentage of public internet traffic, maybe you need to be held to a different standard and set of rules than a startup trying to get a foothold among…

The three other replies you've gotten so far have given some generically applicable though still good answers, but I want to address something regard Cloudflare specifically: a major part of their entire core goal and value proposition revolves around being able to defend their customers from continuously scaling ever larger hostile attacks. This isn't merely a case of "natural selection" or what a company/VCs might desire, but that it's hard to see how under the current (depressing, shitty) state of the Internet it'd be possible to cheaply defend against terabit-plus class DDOS and the like without Cloudflare level scale in turn. And "cheaply" is in fact critical too because the whole point of resource exhaustion attacks is that they're purely economic, if it costs many times more to mitigate them then to launch and profit from them then the attackers are going to win in the end. Ideally we'd be solving this collective action problem collectively with standards amongst nations and ISPs to mitigate or eliminate botnets at the source, but we have to trundle along as best we can in the mean time right? I'm not sure there is room for a large number of players in Cloudflare's role, and they've been a pretty dang decent one so far.

Re: Cloudflare outage should not have happened

#69

Post by parados »

tptacek wrote: post #10

Cloudflare doesn't seem to have called it a "Root Cause Analysis" and, in fact, the term "root cause" doesn't appear to occur in Prince's report. I bring this up because there's a school of thought that says "root cause analysis" is counterproductive: complex systems are always balanced on the precipice of multicausal failure.

True, and I agree, but from their report they do seem to be doing Root Cause Analysis (RCA) even if they don't call it that.

RCA is a really bad way of investigating a failure. Simply put; if you show me your RCA I know exactly where you couldn't be bothered to look any further.

I think most software engineers using RCA confuse the "cause" ("Why did this happen") with the solution ("We have changed this line of code and it's fixed"). These are quite different problem domains.

Using RCA to determine "Why did this happen" is only useful for explaining the last stages of an accident. It focuses on cause->effect relationships and tells a relatively simple story but one that is easy to communicate - Hi there managers and media! But RCA only encourages simple countermeasures which will probably be ineffective and will be easily outrun by the complexity of real systems

However one thing RCA is really good at is allocating blame. If your organisation is using RCA then, what ever you pretend, your organisation has a culture of blame. With a blame culture (rather than a reporting culture) your organisation is much more likely to fail again. You will lack operational resilience.

Re: Cloudflare outage should not have happened

#70

Post by hnthrowaway0328 »

nine_k wrote: post #9

* The unwrap() in production code should have never passed code review. Damn, it should have been flagged by a linter. * The deployment should have followed the blue/green pattern, limiting the blast radius of a bad change to a subset of nodes. * In general, a company so much at the foundational level of internet connectivity should not follow the "move fast, break things" pattern. They did not have an overwhelming r…

I wish they do burn a lot of trust to show up in their financial reports. Otherwise it is like "we do not like it but gonna use it anyway".
Post reply on HN