Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

111–120 of 265 posts

Re: Cloudflare outage should not have happened

#111

Earlier quoted context omitted.

> I’ve been seeing you blazing this trail since the incident and it feels a short sighted and reductive. Why is it inappropriate to be able to statically label the behavior? Maybe I don't want my failure behavior dictated by a downstream dependency or distracted engineer. The subject of how to fail is a big topic and is completely orthogonal to the topic of how can we know about this and shape our outcomes. I would r…

How deep do you go? Being forced to label any function that allocates memory with ”panic”? Right now you all the instances where the code can panic are labeled. Grep for unwrap, panic, expect etc. In all my years of professional Rust development I’ve never seen a potential panic pass code review without a discussion. Unless it was trivial like trying to build an invalid Regex from a static string.

Malloc is fair game.

Unwrap, slice access, etc. are not.

Re: Cloudflare outage should not have happened

#112

Earlier quoted context omitted.

I laughed out loud when he said Cloudflare should have formally verified its systems.

Not to single you out in particular, but I see this sentiment among programmers a lot and to me it's akin to a structural engineer saying "I laughed out loud when he said they should analyze the forces in the bridge".

You can't formally verify anything that uses consensus, which is the backbone of the entire web. It's a complete non-starter.

Re: Cloudflare outage should not have happened

#113

Earlier quoted context omitted.

How deep do you go? Being forced to label any function that allocates memory with ”panic”? Right now you all the instances where the code can panic are labeled. Grep for unwrap, panic, expect etc. In all my years of professional Rust development I’ve never seen a potential panic pass code review without a discussion. Unless it was trivial like trying to build an invalid Regex from a static string.

Malloc is fair game. Unwrap, slice access, etc. are not.

And now the endless bikeshedding has begun.

Thanks for making abundantly clear how such a feature wouldn’t solve a thing.

Re: Cloudflare outage should not have happened

#114
I have to disagree on the tests not potentially helping here. Finding the right abstraction layer is hard, but there was obviously no integration test that tested wherever the original query was being constructed and where the output was being used. A single smoke test would have failed the same way their actual infra failed when the change was introduced.

Obviously, that's not to say that writing normalized database schemas and formal specification won't reduce the number of problems you will introduce. But people make mistakes anywhere, which could have been the case here with the query even if the DB was in a NF (and it still could have been in their case), or in the formal spec as well.

There is no magic bullet for correctness, unfortunately.

Re: Cloudflare outage should not have happened

#115
post #34

Hindsight bias is always easier but: > FAANG-style companies are unlikely to adopt formal methods or relational rigor wholesale. But for their most critical systems, they should. It’s the only way to make failures like this impossible by design, rather than just less likely. That relational rigor imposes what one chooses to be true, it isn’t a universal truth. The frame problem and the qualification problem apply her…

I expect the downvotes here but it is important.

It doesn’t matter if you get there through Trakhtenbrot or Rice.

Codd’s normal form is a projection, it will turn your fancy model logic into classic logic.

IMHO it is always something to look for to use as a default, but fails if it is a hard requirement.

One classic way to describe the problem is the White king and Alice.

> ‘I see nobody on the road,’ said Alice.

> ‘I only wish I had such eyes,’ the King remarked in a fretful tone. ‘To be able to see Nobody! And at that distance, too! Why, it’s as much as I can do to see real people, by this light!’

Codd added nulls to handle unknowns or missing data.

The proper use of them is a complex subject. But they are required if you care about semantic correctness and not just logical validity in many cases.

Diaconescu-Goodman-Myhill theorem[0] will show the equivalence between PEM, finite indexes, and choice

[0] https://ncatlab.org/nlab/show/Diaconescu-Goodman-Myhill+theo...

Re: Cloudflare outage should not have happened

#116

Earlier quoted context omitted.

I laughed out loud when he said Cloudflare should have formally verified its systems.

Not to single you out in particular, but I see this sentiment among programmers a lot and to me it's akin to a structural engineer saying "I laughed out loud when he said they should analyze the forces in the bridge".

A bridge failing is a high likelihood of death or serious injury. How many people died or were seriously injured in the latest Cloudflare outage?

For life or death systems, I agree that we should be looking to implement analogous processes/systems to a structural engineer or doctor, etc. Cloudflare is not a life or death system. If you operate a life or death system and you have Cloudflare as a single point of failure, for some reason, that should not be Cloudflare's problem.

Re: Cloudflare outage should not have happened

#117
post #71

Earlier quoted context omitted.

Rust needs to get rid of .unwrap() and its kin. They're from pre-1.0 Rust, before many of the type system features and error handling syntax sugar were added. There's no reason to use them as the language provides lots of safer alternatives. If you do want to trigger a panic, you can, but I'd also ask - why? Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other…

> There's no reason to use [panics] as the language provides lots of safer alternatives. Dunno ... I think runtime assertions and the ability to crash a misbehaving program are a pretty important part of the toolset. If rust required `Result`s to be wired up up and down the entire call tree for the privilege of using a runtime assertion, I think it would be a lot less popular, and probably less safe in practice. > Al…

> I 100% agree that a mechanism to prove that code can or cannot panic would be great, but why would malloc be special here? Folks who are serious about preventing panics will generally use `no-std` in order to prevent malloc in the first place.

In one of the domains I work in, a malloc failure and OOMkill are equivalent. We just restart the container. I've done all the memory pressure measurement ahead of time and reasonably understand how the system will behave under load. Ideally it should never happen because we pay attention to this and provision with lots of overhead capacity, failover, etc. We have slow spillover rather than instantaneous catastrophe. Then there's instrumentation, metrics, and alerting.

A surprise bug in my code or a dependency that causes an unexpected panic might cause my application or cluster to restart in ways we cannot predict or monitor. And it can happen across hundreds of application instances all at once. There won't be advanced notice, and we won't have a smoking gun. We might waste hours looking for it. It could be as simple as ingesting a pubsub message and calling unwrap(). Imagine an critical service layer doing this all at once, which in turn kills downstream services, thundering herds of flailing services, etc. - now your entire company is on fire, everyone is being paged, and folks are just trying to make sense of it.

The fact is that the type of bugs that might trigger a user-induced panic might be hidden for a long time and then strike immediately with millions of dollars of consequences.

Maybe the team you implemented an RPC for six months ago changes their message protocol by flipping a flag. Or maybe you start publishing keys with encoded data center affinity bytes, but the schema changed, and the library that is supposed to handle routing did an unwrap() against a topology it doesn't understand - oops! Maybe the new version handles it, but you have older versions deployed that won't handle it gracefully.

These failures tend to sneak up on you, then happen all at once, across the entire service, leaving you with no redundancy. If you ingest a message that causes every instance to death spiral, you're screwed. Then you've got to hope your logging can help you find it quickly. And maybe it's not a simple roll back to resolve. And we know how long Rust takes to build...

The best tool for this surely can't be just a lint? In a supposedly "safe" language? And with no way to screen dependencies?

Just because somebody's use case for Rust is okay with this behavior doesn't mean everyone's tolerates this. Distributed systems folks would greatly appreciate some control over this.

All I'm asking for is tools to help us minimize the surface area for panics. We need as much control over this as we can get.

Re: Cloudflare outage should not have happened

#118

Earlier quoted context omitted.

Malloc is fair game. Unwrap, slice access, etc. are not.

And now the endless bikeshedding has begun. Thanks for making abundantly clear how such a feature wouldn’t solve a thing.

https://news.ycombinator.com/item?id=46060907

Copying this so you see it too -

The Cloudflare outage was a multi-billion dollar outage. I have personally been involved in multiple hundred million dollar outages at fintechs, so forgive me for being passionate about this.

Several of the outages I've been involved in were the result of NPEs or incorrectly processing runtime data. Rust has tools to enforce safety here, but it doesn't have tools to enforce your use of them. If also doesn't have a way to safeguard you from others deciding the behavior for you.

There is potentially a very easy set of non-onerous features we could build that allow us to prevent this.

Re: Cloudflare outage should not have happened

#119

Earlier quoted context omitted.

When you're powering this large a fraction of the internet is it even an option not to work like that? You'd think that with that kind of market cap resource constraints should no longer be holding you back from doing things properly.

I work in formal verification at a FAANG. It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths, and even then, the properties asserted by formal verification are often quite a bit less powerful than necessary to truly guarantee something useful. I want formal verification everywhere. I believe in provable correctness. I…

We need modern programming languages with formal verification built-in - should be applicable to specially demarcated functions/modules. It is a headache to write TLA+ and keep the independent spec up2date with the productive code.

Re: Cloudflare outage should not have happened

#120

Earlier quoted context omitted.

How deep do you go? Being forced to label any function that allocates memory with ”panic”? Right now you all the instances where the code can panic are labeled. Grep for unwrap, panic, expect etc. In all my years of professional Rust development I’ve never seen a potential panic pass code review without a discussion. Unless it was trivial like trying to build an invalid Regex from a static string.

Malloc is fair game. Unwrap, slice access, etc. are not.

So slicing is forbidden in this scheme? But not malloc?

This doesn’t seem to be a principled stance on making the language safer. It feels a bit whack-a-mole. “Unwrap is pretty easy to give up. I could live without slicing. Malloc seems hard though. I don’t want to give that up.”

Post reply on HN