Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

Post reply on HN
Print view View original

241–250 of 265 posts

Re: Cloudflare outage should not have happened

#241

Post by jacquesm »

joshuamorton wrote: post #91
jacquesm wrote: post #77

Earlier quoted context omitted.

You are simplifying the control software of an insulin point to a degree that does not match reality. I'm saying that because I actually reviewed the code of one and the amount of safety consciousness on display there was off the charts compared to what you usually encounter in typical web development. You also under-estimate the dynamic nature of the environment these pumps operate in as well as the amount of contin…

I had a former coworker who moved from the medical device industry to similar-to-cloudflare-web software. While he had some appreciation for the validation and intense QA they did (they didn't use formal methods, just heavy QA and deep specs), it became very clear to him very clearly that those approaches don't work with speed-of-release as a concern (his development cycles were annual, not weekly or daily). And they…

It is perfectly possible to engineer for faster cycles without losing control over what your code can and can not do. It is harder, for sure. But I do not think it is a matter of this absolutely not working, that's black-and-white and it never is black and white, it is always some shade of gray.

For instance: validating a configuration before loading it is fairly standard practice, as are smoke tests and gradual roll-outs. Configuration fuck-ups are fairly common so you engineer with that in mind.

Re: Cloudflare outage should not have happened

#242

Post by rcxdude »

DrSusanCalvin wrote: post #95
engineeringwoke wrote: post #86

Earlier quoted context omitted.

I laughed out loud when he said Cloudflare should have formally verified its systems.

Not to single you out in particular, but I see this sentiment among programmers a lot and to me it's akin to a structural engineer saying "I laughed out loud when he said they should analyze the forces in the bridge".

more like "I laughed out loud when he said they should FEM the whole structure, down to the last bolt and strand of cable".

(More seriously, 'formal verification' is not a single thing, more a class of techniques which allow you to statically guarantee some properties of the system you are working with. When you propose it, you should have a clear idea of what properties you care about and how you intend to prove them, as well as a strong concern about whether those properties are actually going to capture enough of what you care about for it to be worthwhile)

Re: Cloudflare outage should not have happened

#243

Post by jacquesm »

Dylan16807 wrote: post #205
gishh wrote: post #197

Earlier quoted context omitted.

What about “detect when the content isn’t correct and take protective measures so that a core service of the global internet _doesn’t_ crash?” Wasn’t that the whole point of rust? I’ll repeat again “if it compiles it is almost absolutely correct” is a mantra I see on hn daily. Apparently that isn’t true. Edit: isn’t the whole idea of C/C++ being flawed pivoted around memory management and how flawed the languages are…

If you corrupt memory, a huge variety of unpredictable bad things can happen. If you exit, a known bad thing happens. No language can protect you from a program's instructions being broken. What protective measures do you have in mind? Do they still result in the service ceasing to process data and reporting a problem to the central controller? The difference between "stops working and waits" and "stops working and c…

Ok, I'll take a stab at that:

I would expect such a critical piece of code to be able to hot-load and validate a new configuration before it is put into action. I would expect such a change to be rolled out gradually, or at least as gradually as required to ensure that it functions properly before it is able to crash the system wholesale.

I can't say without a lot more knowledge about the implementation and the context what the best tools would be to achieve this but I can say that crashing a presently working system because of a config fuckup should not be in the range of possible expected outcomes.

Because config fuckups are a fact of life so config validation before release is normal.

Re: Cloudflare outage should not have happened

#244

Post by drob518 »

bigstrat2003 wrote: post #227
mikestorrent wrote: post #187

Earlier quoted context omitted.

[flagged]

This has nothing to do with the language, and it's so irritating to see people falsely claiming it is. There is nothing whatsoever about Rust that meant the engineer had to write code to the effect of if result.is_err() { panic!() } That was a choice on the engineer's part, not something caused by the language. You could choose to write that code in any language. It might even be the right choice sometimes! But wheth…

Agreed. A tool may allow the programmer to do something (with varying degrees of difficulty), but it’s always the programmer’s choice. Tools are inert by themselves. Only humans make choices.

Re: Cloudflare outage should not have happened

#245

Post by drob518 »

trillic wrote: post #158
dpark wrote: post #63

Earlier quoted context omitted.

> we know how to do it better, but nearly all of even the smartest engineers we can hire are not smart enough. This seems like a contradiction. If the smartest engineers you can hire are not smart enough to work within formal verification constraints then we in fact do not know how to do this. If formal verification hinges on having perfect engineers then it’s useless because perfect engineers wouldn’t need formal ve…

It’s not that we can’t do it, it’s that higher-velocity occasionally buggy code has proven time and time again to be significantly more profitable than formally verified. The juice is rarely worth the squeeze.

Agreed. Further, this has been true even ignoring formal verification. Who has been in the situation of making the choice to ship known-buggy code to make a release date or satisfy a customer demand for other functionality? All of us, I suspect, if we’re being honest. I certainly have.

Re: Cloudflare outage should not have happened

#246

Post by drob518 »

Nevermark wrote: post #201
dpark wrote: post #144

Earlier quoted context omitted.

If the smartest engineers cannot do it, it doesn’t work. This isn’t even getting to the practical question of whether it’s worth doing, given the significant additional cost. If the smartest folks you can find are not smart enough to use the framework then it’s useless. Maybe this means the tooling is insufficient. Maybe it means the field isn’t mature enough. Whatever, if you need an IQ two standard deviations above…

> If the smartest engineers cannot do it, it doesn’t work. [FIX:] ..., it doesn't work universally. And the answer to that is pretty clear. It does not work universally. If every developer started only shipping code they had credibly formally verified, the vast majority of developers would go into shock at the scale of work to be done. Even the best "validators" would fall into career shredding pits, due to "minor" b…

> But formal validation can still work some of the time with the right people, on the right scale and kind of project, with the right amount of resources/time expended.

The problem is, it’s unclear exactly what those situations are or even should be. That lack of clarity causes us to fail to recognize when we could have applied these methods and so we just don’t. As much as I see value in formal methods, I’ve never worked with a team that has employed them. And I don’t think I’m at all unique in that.

Re: Cloudflare outage should not have happened

#247

Post by jacquesm »

hdgvhicv wrote: post #239
jacquesm wrote: post #25

Earlier quoted context omitted.

When you're powering this large a fraction of the internet is it even an option not to work like that? You'd think that with that kind of market cap resource constraints should no longer be holding you back from doing things properly.

The bug the fix here is the “powering a large fraction of the internet” The lack of diversity is a major problem.

the -> to ?

And yes, agreed.

Re: Cloudflare outage should not have happened

#248

Post by frumplestlatz »

dragonwriter wrote: post #226
frumplestlatz wrote: post #225

Earlier quoted context omitted.

“Narrowing” a compile-time invariant without a corresponding proof is formally unsound and does not “respect” the declared invariant in any reasonable sense. What’s silly is the desire to pretend otherwise because it’s easier.

> “Narrowing” a compile-time invariant without a corresponding proof is formally unsound and does not “respect” the declared invariant in any reasonable sense The invariant is that either condition X applies or condition Y applies. "Panic and stop execution if X, continue execution with the invariant Y if Y" is not unsound and does respect the original invariant in every possible sense. It may be the wrong choice of…

Claiming panic as sound and not a type-level problem is very cute, but also clearly wrong and a bit hilarious after the outage in question.

You guys really will go to any possible rhetorical length to justify lazy programming practices in error handling.

Re: Cloudflare outage should not have happened

#249

Post by burntsushi »

jacquesm wrote: post #243
Dylan16807 wrote: post #205

Earlier quoted context omitted.

If you corrupt memory, a huge variety of unpredictable bad things can happen. If you exit, a known bad thing happens. No language can protect you from a program's instructions being broken. What protective measures do you have in mind? Do they still result in the service ceasing to process data and reporting a problem to the central controller? The difference between "stops working and waits" and "stops working and c…

Ok, I'll take a stab at that: I would expect such a critical piece of code to be able to hot-load and validate a new configuration before it is put into action. I would expect such a change to be rolled out gradually, or at least as gradually as required to ensure that it functions properly before it is able to crash the system wholesale. I can't say without a lot more knowledge about the implementation and the conte…

Yes, we are lacking information.

But the operative point in this sub thread is whether unwrap() specifically is load bearing.

If instead they bubbled up the error, printed it and then exited the program---without ever using unwrap---then presumably they still would have had a denial of service problem as a result of OOM.

And even if unwrap were load bearing here, then we would be in agreement that it was an inappropriate use of unwrap. But we are still nowhere near saying "unwrap should literally never be used in production."

Re: Cloudflare outage should not have happened

#250

Post by pas »

locknitpicker wrote: post #4

This sort of Monday morning quarterbacking is pointless and only serves as a way for random bloggers to try to grab credit without actually doing or creating any value.

pointing out that the basics matter is a valuable insight

if this comes with the side-effect of some random blogger getting "easy" credit then so be it

the same thing happens in economics, where the bitter lessons have to be learned again and again, and nowadays in politics (and even in ethics) too

and of course here people very correctly immediately started talking about the trade-offs involved in making sure that these aforementioned basics are really taken care of, and whether are there low-hanging fruits (does having a better DB schema provide enough benefit for it to worth it? or do you need to couple it with this or that compiler too? but isn't that simply too much? when the critical infrastructure provider should transition to the mindset that most people expect from a critical infrastructure provider? and so on...)

whereas in ethics (and intersectional politics) for example there's an unfortunate bitter denial of trade offs

Post reply on HN