Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

Post reply on HN
Print view View original

91–100 of 265 posts

Re: Cloudflare outage should not have happened

#91

Post by joshuamorton »

jacquesm wrote: post #77
simlevesque wrote: post #38

Earlier quoted context omitted.

Yeah but the anti-DDOS feature needs to react to new methods all the time, it's not a static thing you build once and it works forever. An insulin pump is very different. Your human body, insulin, and physics aren't changing any time soon.

You are simplifying the control software of an insulin point to a degree that does not match reality. I'm saying that because I actually reviewed the code of one and the amount of safety consciousness on display there was off the charts compared to what you usually encounter in typical web development. You also under-estimate the dynamic nature of the environment these pumps operate in as well as the amount of contin…

I had a former coworker who moved from the medical device industry to similar-to-cloudflare-web software. While he had some appreciation for the validation and intense QA they did (they didn't use formal methods, just heavy QA and deep specs), it became very clear to him very clearly that those approaches don't work with speed-of-release as a concern (his development cycles were annual, not weekly or daily). And they absolutely don't work in contexts where user-abuse or reactivity are necessary. The contexts are just totally different.

Re: Cloudflare outage should not have happened

#92

Post by yearolinuxdsktp »

vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

Not to mention that perfectly normalizing a database always incurs join overhead that limits horizontal scalability. In fact, denormalization is required to achieve scale (with a trade-off).

I’m not sure how formal verification would’ve prevented this issue from happening. In my experience, it’s unusual to have to specify a database name in the query. How could have formal verification covered this outcome?

The recommendations don’t make sense saying that the query needed DISTINCT and LIMIT. Don’t forget that the incoming data was different (r0 and default did not return the same exact data, this is why the config files more than doubled in size), so using DISTINCT would have led to uncertain blending of data, producing neither result and hiding the double-database read altogether. Secondly, LIMIT only makes sense to use in conjunction with a failure circuit breaker (if LIMIT items is returned, fail the query). When does it make business-logic sense to LIMIT the query-in-question’s result? And do you think the authors would have known how to set the LIMIT to not exceed the configuration file consumers’ limitations?

The article says: > “You can’t reliably catch that with more tests or rollouts or flags. You prevent it by construction—through analytical design.”

That’s the big design up front fallacy. Of course you can catch it reliably with more tests, and limit the damage with flags and rollouts. There’s zero guarantee that the analytical design would’ve caught this up front.

Re: Cloudflare outage should not have happened

#93

Post by ViewTrick1002 »

echelon wrote: post #71
frumplestlatz wrote: post #47

Earlier quoted context omitted.

Agreed. I left out any commentary on `.unwrap()` from my original comment, but it’s an obvious example of something that should never have appeared in critical code.

Rust needs to get rid of .unwrap() and its kin. They're from pre-1.0 Rust, before many of the type system features and error handling syntax sugar were added. There's no reason to use them as the language provides lots of safer alternatives. If you do want to trigger a panic, you can, but I'd also ask - why? Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other…

I’ve been seeing you blazing this trail since the incident and it feels short sighted and reductive.

Rust is built on forcing the developer to acknowledge the complexity of reality. Unwrap acknowledges said complexity with a perfectly valid decision.

There are a few warts from early days like indexing and the ”as” operator where the easy path is doing the wrong thing.

But unwraps or expects are where Rust shines. Throwing up your hands is a perfectly reasonable response.

With your approach, what should Cloudflare have done?

Return an error, log it and return a 500 result due to invalid config? They could fail open, but then that opens another enormous can of worms.

There simply are no good options.

The issue rests upstream where deployments and effects between disparate services needs to be mapped and managed.

Which is a truly hard problem, rather than blaming the final piece throwing up its hand when given an invalid config.

Re: Cloudflare outage should not have happened

#94

Post by ViewTrick1002 »

jacquesm wrote: post #80
echelon wrote: post #71

Earlier quoted context omitted.

Rust needs to get rid of .unwrap() and its kin. They're from pre-1.0 Rust, before many of the type system features and error handling syntax sugar were added. There's no reason to use them as the language provides lots of safer alternatives. If you do want to trigger a panic, you can, but I'd also ask - why? Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other…

I'd say the equivalent of Erlang's supervisor trees is what is needed but once you go that route you might as well use Erlang.

Or just deploy containers with an orchestrator restarting them when failing?

It is not like an Erlang service would be able to make progress with an invalid config either.

Re: Cloudflare outage should not have happened

#95

Post by DrSusanCalvin »

engineeringwoke wrote: post #86
locknitpicker wrote: post #14

Earlier quoted context omitted.

> I disagree. I learnt good stuff from this article and it’s enough. That's perfectly fine. It's also besides the point though. You can learn without reading random people online cynically shit talking others as a self promotion strategy. This is junior dev energy manifesting junior level understanding of the whole problem domain. There's not a lot to learn from claims that boil down to "don't have bugs".

I laughed out loud when he said Cloudflare should have formally verified its systems.

Not to single you out in particular, but I see this sentiment among programmers a lot and to me it's akin to a structural engineer saying "I laughed out loud when he said they should analyze the forces in the bridge".

Re: Cloudflare outage should not have happened

#96

Post by locknitpicker »

galleywest200 wrote: post #67
locknitpicker wrote: post #14

Earlier quoted context omitted.

> I disagree. I learnt good stuff from this article and it’s enough. That's perfectly fine. It's also besides the point though. You can learn without reading random people online cynically shit talking others as a self promotion strategy. This is junior dev energy manifesting junior level understanding of the whole problem domain. There's not a lot to learn from claims that boil down to "don't have bugs".

> You can learn without reading random people online Somebody has to write something in the first place for one to learn from it, even if the writing is disagreeable.

You failed to cite the comment you were replying to.

The comment is:

> You can learn without reading random people online cynically shit talking others as a self promotion strategy.

Re: Cloudflare outage should not have happened

#97

Post by DrSusanCalvin »

locknitpicker wrote: post #14
nmoura wrote: post #6

Earlier quoted context omitted.

I disagree. I learnt good stuff from this article and it’s enough.

> I disagree. I learnt good stuff from this article and it’s enough. That's perfectly fine. It's also besides the point though. You can learn without reading random people online cynically shit talking others as a self promotion strategy. This is junior dev energy manifesting junior level understanding of the whole problem domain. There's not a lot to learn from claims that boil down to "don't have bugs".

This article actually explains how this bug in particular could have been avoided. Sure you may not consider his approach realistic, but it's not at all saying "don't have bugs". In fact, not having formal verification or similar tooling in place, would be more like saying "just don't write buggy code".

Re: Cloudflare outage should not have happened

#98

Post by no_wizard »

vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

Why is this inherently slower?

There’s for example, languages or features of languages that work entirely on not allowing these things.

I ask because I feel like I’m missing something

Re: Cloudflare outage should not have happened

#99

Post by swiftcoder »

vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

> It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically.

We could also invest in tooling to make this kind of thing easier. Unclear why humans need to hand-normalise the database schema - isn't this exactly the kind of thing compilers are good at?

Post reply on HN