Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

Post reply on HN
Print view View original

231–240 of 265 posts

Re: Cloudflare outage should not have happened

#231

Post by bigstrat2003 »

Ysraes wrote: post #195
echelon wrote: post #43

Earlier quoted context omitted.

unwrap() and the family of methods like it are a Rust anti-pattern from the early days of Rust. It dates back to before many of the modern error-handling and safety-conscious features of the language and type system. Rust is being pulled in so many different directions from new users that the language perhaps never originally intended. Some engineers will be fine with panicky behavior, but a lot of others want to be…

This is starting to sound a lot like checked exceptions in Java.

Checked exceptions were a great idea which are still, to this day, unfairly maligned.

Re: Cloudflare outage should not have happened

#233

Post by sfink »

echelon wrote: post #118
ViewTrick1002 wrote: post #113

Earlier quoted context omitted.

And now the endless bikeshedding has begun. Thanks for making abundantly clear how such a feature wouldn’t solve a thing.

https://news.ycombinator.com/item?id=46060907 Copying this so you see it too - The Cloudflare outage was a multi-billion dollar outage. I have personally been involved in multiple hundred million dollar outages at fintechs, so forgive me for being passionate about this. Several of the outages I've been involved in were the result of NPEs or incorrectly processing runtime data. Rust has tools to enforce safety here, b…

Except that the outage would still have happened without that .unwrap(). So go ahead and build those features, they sound useful, but don't think that they'd save you from a failure like this.

As the poster here said, the place to build in features that would have prevented this from happening is the DB schema and queries. 5NF would be onerous overkill here, but it seems reasonable to have some degree of forced normalization for something that could affect this much.

(Requiring formal verification of everything involved here would be overkilling the overkill, otoh.)

Re: Cloudflare outage should not have happened

#234

Post by Ysraes »

bigstrat2003 wrote: post #231
Ysraes wrote: post #195

Earlier quoted context omitted.

This is starting to sound a lot like checked exceptions in Java.

Checked exceptions were a great idea which are still, to this day, unfairly maligned.

I don't necessarily disagree, I wasn't making a judgement positively or negatively.

Re: Cloudflare outage should not have happened

#235

Post by jacquesm »

MichaelZuo wrote: post #199
jacquesm wrote: post #77

Earlier quoted context omitted.

You are simplifying the control software of an insulin point to a degree that does not match reality. I'm saying that because I actually reviewed the code of one and the amount of safety consciousness on display there was off the charts compared to what you usually encounter in typical web development. You also under-estimate the dynamic nature of the environment these pumps operate in as well as the amount of contin…

If humans beings had a small chance to transform into say quadrupeds or suddenly grow tenatacles, extra hearts, organs, etc., in any given year… then wouldn’t designing a safe insulin pump literally be impossible?

Compared to what they are already doing it would be marginally more difficult.

Re: Cloudflare outage should not have happened

#236

Post by DrSusanCalvin »

orionometer wrote: post #116
DrSusanCalvin wrote: post #95

Earlier quoted context omitted.

Not to single you out in particular, but I see this sentiment among programmers a lot and to me it's akin to a structural engineer saying "I laughed out loud when he said they should analyze the forces in the bridge".

A bridge failing is a high likelihood of death or serious injury. How many people died or were seriously injured in the latest Cloudflare outage? For life or death systems, I agree that we should be looking to implement analogous processes/systems to a structural engineer or doctor, etc. Cloudflare is not a life or death system. If you operate a life or death system and you have Cloudflare as a single point of failur…

Sure, but this sentiment is why software "engineering" isn't really. You can justify it by not being important enough for actual engineering practices I guess, but to me it's a lack of pride in and care of your product.

Re: Cloudflare outage should not have happened

#237

Post by DrSusanCalvin »

locknitpicker wrote: post #150
DrSusanCalvin wrote: post #97

Earlier quoted context omitted.

This article actually explains how this bug in particular could have been avoided. Sure you may not consider his approach realistic, but it's not at all saying "don't have bugs". In fact, not having formal verification or similar tooling in place, would be more like saying "just don't write buggy code".

> This article actually explains how this bug in particular could have been avoided. Not really. The article is a textbook example of hindsight bias. It's a simplistic analysis of a far more complex problem that goes over the blogger's head, and results in a string of simplistic assertions that fail to address any of the issues. Read up on the definition of monday morning quarterback.

Read up on the value of snarky and dismissive comments spouting simplistic cliches.

Re: Cloudflare outage should not have happened

#238

Post by jacquesm »

syockit wrote: post #216
jacquesm wrote: post #77

Earlier quoted context omitted.

You are simplifying the control software of an insulin point to a degree that does not match reality. I'm saying that because I actually reviewed the code of one and the amount of safety consciousness on display there was off the charts compared to what you usually encounter in typical web development. You also under-estimate the dynamic nature of the environment these pumps operate in as well as the amount of contin…

The point still stands. The human body still isn't going change. That's why insulin pump can afford to have all kinds of rigorous engineering, while web-facing infrastructure on the other hand needs to be able to quickly adapt to changes.

> That's why insulin pump can afford to have all kinds of rigorous engineering, while web-facing infrastructure on the other hand needs to be able to quickly adapt to changes.

The only reason we have a web in the first place is because of rigorous engineering. The whole thing was meant to be decentralized, if you're going to purposefully centralize a critical feature you are not going to get away with 'oh we need to quickly adapt to changes so let's abandon rigor'.

That's just irresponsible. In that case we'd be better off without CF. And I don't see CF arguing this, in fact I'm pretty sure that CF would be more than happy to expend the extra cycles so maybe stop attempting to make them look bad?

Re: Cloudflare outage should not have happened

#239

Post by hdgvhicv »

jacquesm wrote: post #25
vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

When you're powering this large a fraction of the internet is it even an option not to work like that? You'd think that with that kind of market cap resource constraints should no longer be holding you back from doing things properly.

The bug the fix here is the “powering a large fraction of the internet”

The lack of diversity is a major problem.

Post reply on HN