Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

121–130 of 265 posts

Re: Cloudflare outage should not have happened

#121
post #120

Earlier quoted context omitted.

Malloc is fair game. Unwrap, slice access, etc. are not.

So slicing is forbidden in this scheme? But not malloc? This doesn’t seem to be a principled stance on making the language safer. It feels a bit whack-a-mole. “Unwrap is pretty easy to give up. I could live without slicing. Malloc seems hard though. I don’t want to give that up.”

I posted about why this is important for distributed systems engineering:

https://news.ycombinator.com/item?id=46060907

Malloc is fine. We can and do monitor that. It's these undetectable runtime logic problems that are land mines.

In distributed systems, these can cause contagion and broad outages. Recovering can be very difficult and involve hours of complex steps across dozens of teams. Meanwhile you're losing millions, or even hundreds of billions, of dollars for you and your customers.

Someone unwrapping() a Serde wire message or incorrectly indexing a payload should not cause an entire fleet to crash. The tools should require the engineer handle these problems with language features such as Result.

Presently, who knows if your downstream library dependency unwrap()s under the hood?

This is a big deal and there could be a very simple and effective fix.

The Cloudflare outage was a multi-billion dollar outage. I have personally been involved in multiple hundred million dollar outages at fintechs, so forgive me for being passionate about this.

Re: Cloudflare outage should not have happened

#122

Earlier quoted context omitted.

I'd say the equivalent of Erlang's supervisor trees is what is needed but once you go that route you might as well use Erlang.

Or just deploy containers with an orchestrator restarting them when failing? It is not like an Erlang service would be able to make progress with an invalid config either.

What happens when they "keep" failing ? You never get to know what is causing your nightmare.

Re: Cloudflare outage should not have happened

#123

Earlier quoted context omitted.

Not to single you out in particular, but I see this sentiment among programmers a lot and to me it's akin to a structural engineer saying "I laughed out loud when he said they should analyze the forces in the bridge".

You can't formally verify anything that uses consensus, which is the backbone of the entire web. It's a complete non-starter.

Care to elaborate? Perhaps the tools to do this in practice aren't there (which just shows how young the field of software "engineering" really is), but what consensus are you talking about and how is it an obstacle to verifying code? Most of the web follows standards and protocols, which actually sort of a prerequisite for communications across different systems...

Re: Cloudflare outage should not have happened

#124
TFA has a point that it should never have happened, and that CF software engineering practices are likely to blame.

But a BCNF (or 5NF or whatever) database without nullable columns wouldn't have prevented it. Formally verified code might have but that remains a pipe dream for any significant code base.

The proposed cure is worse than the disease.

Re: Cloudflare outage should not have happened

#125

Earlier quoted context omitted.

I'd say the equivalent of Erlang's supervisor trees is what is needed but once you go that route you might as well use Erlang.

Or just deploy containers with an orchestrator restarting them when failing? It is not like an Erlang service would be able to make progress with an invalid config either.

That's fair, but even there the roll-back would be a lot smoother, besides the supervisor trees are a lot more fine grained than restarting entire containers when they fail.

Re: Cloudflare outage should not have happened

#126

Earlier quoted context omitted.

Ok, let's start off with holding them to the same standards as avionics software development. The formal verification can wait.

Agreed. I left out any commentary on `.unwrap()` from my original comment, but it’s an obvious example of something that should never have appeared in critical code.

ON HN, just a couple of years ago, a famous Rust programmer said that it is OK to use unwrap. Rustaceans supported this position. Cloudflare merely followed the community standard.

Using unwrap() in Rust is Okay

https://news.ycombinator.com/item?id=32385102 https://burntsushi.net/unwrap/

Re: Cloudflare outage should not have happened

#127

Earlier quoted context omitted.

Precisely. This is key infrastructure we're talking about not some kind of webshop.

Yeah but the anti-DDOS feature needs to react to new methods all the time, it's not a static thing you build once and it works forever. An insulin pump is very different. Your human body, insulin, and physics aren't changing any time soon.

All the more reason to be careful about relying on humans to avoid making mistakes when changing it rather than moving quickly and letting things fail in production.

Re: Cloudflare outage should not have happened

#128

Earlier quoted context omitted.

You can't formally verify anything that uses consensus, which is the backbone of the entire web. It's a complete non-starter.

Care to elaborate? Perhaps the tools to do this in practice aren't there (which just shows how young the field of software "engineering" really is), but what consensus are you talking about and how is it an obstacle to verifying code? Most of the web follows standards and protocols, which actually sort of a prerequisite for communications across different systems...

Basically the modern web uses orchestration, for pretty much everything. Usually Kubernetes is doing that. Theoretically protocols like RAFT are formally verifiable, but their implementations in orchestration tools like etcd have not been, and I would go so far as to say that that is an impossible task. Therefore, the entire exercise is kind of silly.

Re: Cloudflare outage should not have happened

#129

Earlier quoted context omitted.

Agreed. I left out any commentary on `.unwrap()` from my original comment, but it’s an obvious example of something that should never have appeared in critical code.

ON HN, just a couple of years ago, a famous Rust programmer said that it is OK to use unwrap. Rustaceans supported this position. Cloudflare merely followed the community standard. Using unwrap() in Rust is Okay https://news.ycombinator.com/item?id=32385102 https://burntsushi.net/unwrap/

[deleted]

Re: Cloudflare outage should not have happened

#130

Earlier quoted context omitted.

Ok, let's start off with holding them to the same standards as avionics software development. The formal verification can wait.

Agreed. I left out any commentary on `.unwrap()` from my original comment, but it’s an obvious example of something that should never have appeared in critical code.

[deleted]
Post reply on HN