Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

Post reply on HN
Print view View original

201–210 of 265 posts

Re: Cloudflare outage should not have happened

#201

Post by Nevermark »

dpark wrote: post #144
frumplestlatz wrote: post #134

Earlier quoted context omitted.

> If formal verification hinges on having perfect engineers then it’s useless because perfect engineers wouldn’t need formal verification. It doesn’t hinge on having perfect engineers. It hinges on engineers being able to model problems algebraically and completely, prove the equivalence of multiple such models at different layers of abstraction (including equivalence to the original code), and then prove that useful…

If the smartest engineers cannot do it, it doesn’t work. This isn’t even getting to the practical question of whether it’s worth doing, given the significant additional cost. If the smartest folks you can find are not smart enough to use the framework then it’s useless. Maybe this means the tooling is insufficient. Maybe it means the field isn’t mature enough. Whatever, if you need an IQ two standard deviations above…

> If the smartest engineers cannot do it, it doesn’t work.

[FIX:] ..., it doesn't work universally.

And the answer to that is pretty clear. It does not work universally. If every developer started only shipping code they had credibly formally verified, the vast majority of developers would go into shock at the scale of work to be done. Even the best "validators" would fall into career shredding pits, due to "minor" but now insurmountable dependencies in previously unverified projects. The vast majority of projects would go into unrecoverable stalls.

But formal validation can still work some of the time with the right people, on the right scale and kind of project, with the right amount of resources/time expended.

It isn't as if regular "best practices" programming works universally either. But validation is much harder.

Re: Cloudflare outage should not have happened

#202

Post by asa400 »

echelon wrote: post #111
ViewTrick1002 wrote: post #108

Earlier quoted context omitted.

How deep do you go? Being forced to label any function that allocates memory with ”panic”? Right now you all the instances where the code can panic are labeled. Grep for unwrap, panic, expect etc. In all my years of professional Rust development I’ve never seen a potential panic pass code review without a discussion. Unless it was trivial like trying to build an invalid Regex from a static string.

Malloc is fair game. Unwrap, slice access, etc. are not.

You probably know about these, but for the benefit of folks who don't, you can forbid slice access and direct unwraps with clippy. Obviously this only lints your own code and not dependencies.

  - https://rust-lang.github.io/rust-clippy/master/#unwrap_used
  - https://rust-lang.github.io/rust-clippy/master/#indexing_slicing
  - https://rust-lang.github.io/rust-clippy/master/#string_slice

Re: Cloudflare outage should not have happened

#203

Post by jazzyjackson »

gishh wrote: post #200
saati wrote: post #189

Earlier quoted context omitted.

But it wasn't the culprit, the code could have been in anything, or could have bubbled up errors to main, and it still would have failed with for an incorrect config.

Right. So the language that espoused to eliminate errors that took down large positions of the internet, failed. The specifics matter of course, but the mantra of rust as some safe language that should never have allowed something like this to happen, happened. I vote we rename rust to “rustantic” in honor of human hubris.

I just don't think you have the dunk you think you do. The Rust crowd is very adamant about preventing /many/ bugs. I rarely hear it recommended as a silver bullet that never fails.

Re: Cloudflare outage should not have happened

#204

Post by asa400 »

echelon wrote: post #43
nine_k wrote: post #9

* The unwrap() in production code should have never passed code review. Damn, it should have been flagged by a linter. * The deployment should have followed the blue/green pattern, limiting the blast radius of a bad change to a subset of nodes. * In general, a company so much at the foundational level of internet connectivity should not follow the "move fast, break things" pattern. They did not have an overwhelming r…

unwrap() and the family of methods like it are a Rust anti-pattern from the early days of Rust. It dates back to before many of the modern error-handling and safety-conscious features of the language and type system. Rust is being pulled in so many different directions from new users that the language perhaps never originally intended. Some engineers will be fine with panicky behavior, but a lot of others want to be…

> unwrap() and the family of methods like it are a Rust anti-pattern from the early days of Rust. It dates back to before many of the modern error-handling and safety-conscious features of the language and type system.

I think your argument would be more effective if you dropped this angle.

Unwrap is assert. No more, no less. It's tremendously useful to have it in the language for situations when the cost of encoding some invariant in your program's types is far larger than the benefit you'd gain from doing so. It's not some vestigial thing from way back before anyone received enlightenment that they could use sum types to discriminate errors. It's just a different tool.

I completely agree there are systems and situations where you want to be able to statically verify an absence of panics, but the way you're describing the situation sounds similar to when I hear folks describe anything that came before as "legacy" with a denigrating inflection.

Re: Cloudflare outage should not have happened

#205

Post by Dylan16807 »

gishh wrote: post #197
Dylan16807 wrote: post #192

Earlier quoted context omitted.

In a local sense, "quit out safely when the config is corrupt" is pretty correct. Coordinated systems that test and rollback are way beyond the scope of what a compiler can check.

What about “detect when the content isn’t correct and take protective measures so that a core service of the global internet _doesn’t_ crash?” Wasn’t that the whole point of rust? I’ll repeat again “if it compiles it is almost absolutely correct” is a mantra I see on hn daily. Apparently that isn’t true. Edit: isn’t the whole idea of C/C++ being flawed pivoted around memory management and how flawed the languages are…

If you corrupt memory, a huge variety of unpredictable bad things can happen.

If you exit, a known bad thing happens.

No language can protect you from a program's instructions being broken. What protective measures do you have in mind? Do they still result in the service ceasing to process data and reporting a problem to the central controller? The difference between "stops working and waits" and "stops working and calls abort()" is not much, and usually the latter is preferred because it sets off the alarms faster.

Tell me what specifically you want as correct behavior in this situation.

Re: Cloudflare outage should not have happened

#206

Post by hatthew »

orionometer wrote: post #116
DrSusanCalvin wrote: post #95

Earlier quoted context omitted.

Not to single you out in particular, but I see this sentiment among programmers a lot and to me it's akin to a structural engineer saying "I laughed out loud when he said they should analyze the forces in the bridge".

A bridge failing is a high likelihood of death or serious injury. How many people died or were seriously injured in the latest Cloudflare outage? For life or death systems, I agree that we should be looking to implement analogous processes/systems to a structural engineer or doctor, etc. Cloudflare is not a life or death system. If you operate a life or death system and you have Cloudflare as a single point of failur…

> How many people died or were seriously injured in the latest Cloudflare outage?

I would not be surprised if the answer is "several". The average impact per human is obviously pretty small, but across billions of humans, there will be outliers.

Maybe a fire department uses a coordination system that relies on cloudflare, and with cloudflare down they have to resort to their backup system, and their backup system works but is slightly worse and causes one engine to be delayed in their response, and because they're 3 minutes late, they just miss being able to save someone from the fire.

Maybe someone's running a scientific study on nutrition, and the cloudflare outage means their data collection system is goes down for a bit, so their data flawed, and they end up just barely not passing a some necessary threshold, and they have to rerun their study, and that takes an extra week, and then they miss that quarter's deadline, and then the resulting adjustment to a product/procedure is delayed, and that 3 month delay causes 100,000 people to be slightly more malnourished than they would be otherwise, and one of those people ends up just barely too unhealthy to survive an unrelated deadly illness.

Sure, these scenarios are far-fetched. The chance of if it happening is one-in-a-million.

There are 10000 one-in-a-million people on the earth.

Re: Cloudflare outage should not have happened

#207

Post by otterley »

Analemma_ wrote: post #13
tptacek wrote: post #10

Cloudflare doesn't seem to have called it a "Root Cause Analysis" and, in fact, the term "root cause" doesn't appear to occur in Prince's report. I bring this up because there's a school of thought that says "root cause analysis" is counterproductive: complex systems are always balanced on the precipice of multicausal failure.

When I was at AWS, when we did postmortems on incidents we called it "root cause analysis", but it was understood by everyone that most incidents are multicausal and the actual analyses always ended up being fishbone diagrams. Probably there are some teams which don't do this and really do treat RCA as trying to find a sole root cause, but I think a lot of "getting mad at RCA" is bikeshedding the terminology, and not…

These days we tend to spend more time thinking about the "5 whys" (which often turn into more than 5) than the root cause itself. It's much more productive and useful.

Re: Cloudflare outage should not have happened

#208

Post by femiagbabiaka »

vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

database normalization and formal verification aren't on the same level of difficulty in terms of implementation, and we all could do the former from the beginning, if we choose to (nobody ever chooses to)

Re: Cloudflare outage should not have happened

#209

Post by otterley »

tptacek wrote: post #22
Analemma_ wrote: post #13

Earlier quoted context omitted.

When I was at AWS, when we did postmortems on incidents we called it "root cause analysis", but it was understood by everyone that most incidents are multicausal and the actual analyses always ended up being fishbone diagrams. Probably there are some teams which don't do this and really do treat RCA as trying to find a sole root cause, but I think a lot of "getting mad at RCA" is bikeshedding the terminology, and not…

Right, I'm not a semantic zealot on this point, but the post we're commenting on really does suggest that the Cloudflare incident had a root cause in basic database management failures, which is the substantive issue the root-cause-haters have with the term.

The layered-swiss-cheese model of understanding incidents tends to map to the real world better than the alternatives.

Re: Cloudflare outage should not have happened

#210

Post by uoaei »

vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

> It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically.

Never seen the amoral-capitalist argument to stunting progress at the expense of profit put so succinctly!

Post reply on HN