Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

Post reply on HN
Print view View original

251–260 of 265 posts

Re: Cloudflare outage should not have happened

#251

Post by Dylan16807 »

jacquesm wrote: post #243
Dylan16807 wrote: post #205

Earlier quoted context omitted.

If you corrupt memory, a huge variety of unpredictable bad things can happen. If you exit, a known bad thing happens. No language can protect you from a program's instructions being broken. What protective measures do you have in mind? Do they still result in the service ceasing to process data and reporting a problem to the central controller? The difference between "stops working and waits" and "stops working and c…

Ok, I'll take a stab at that: I would expect such a critical piece of code to be able to hot-load and validate a new configuration before it is put into action. I would expect such a change to be rolled out gradually, or at least as gradually as required to ensure that it functions properly before it is able to crash the system wholesale. I can't say without a lot more knowledge about the implementation and the conte…

> I would expect such a critical piece of code to be able to hot-load and validate a new configuration before it is put into action.

And if that config doesn't validate, what should the process do? Maybe it had a previous config, maybe it didn't. And if it keeps running the old config, that adds extra complication to gradual rollout and makes it harder to understand what state the system is in.

> I would expect such a change to be rolled out gradually, or at least as gradually as required to ensure that it functions properly before it is able to crash the system wholesale.

Me too. Note that doing a gradual rollout doesn't care whether the process uses unwrap or uses something gentler to reject a bad config.

> I can say that crashing a presently working system because of a config fuckup should not be in the range of possible expected outcomes.

By "working system" do you mean the whole thing shouldn't go down, or the single process shouldn't go down? I agree with the former but not the latter.

Re: Cloudflare outage should not have happened

#252

Post by jacquesm »

zbentley wrote: post #219
jacquesm wrote: post #80

Earlier quoted context omitted.

I'd say the equivalent of Erlang's supervisor trees is what is needed but once you go that route you might as well use Erlang.

I’m not sure that panic (speaking generally about the majority of its uses and the spirit of the law; obviously 100% of code does not obey this) is the equivalent of an Erlang process crash in most cases. Rather, I think unwrap()/panic are usually used in ways more similar to erlang:halt/1.

Exactly, but that is kind of the point here. An Erlang 'halt' is something that most Erlang programmers would twig is not what you want in most cases, in most cases you want your process to crash and for the supervisor to restart it if the error is recoverable.

What happened here is systemic: the config file contained an issue severe enough that it precluded the system from running in the first place and unfortunately that caused a runtime error when in fact that validation should have been separate from the actual use. This is where I see the problem with this particular outage. And that makes it an engineering issue much more than a language issue.

Bad configuration files can and do happen, so you take that eventuality into account during systems design.

Re: Cloudflare outage should not have happened

#253

Post by Superuser666 »

igregoryca wrote: post #211
gishh wrote: post #200

Earlier quoted context omitted.

Right. So the language that espoused to eliminate errors that took down large positions of the internet, failed. The specifics matter of course, but the mantra of rust as some safe language that should never have allowed something like this to happen, happened. I vote we rename rust to “rustantic” in honor of human hubris.

The only languages that eliminate logic bugs are formally verified ones, as the article points out. (And even then, your program is only as correct as your specification.) Ordinary Rust code is not formally verified. Anyone who claims Rust eliminates errors is either very naive or lying. Type-safe Rust code is free from certain classes of errors. But that goes out the window the moment you parse input from the outsid…

As someone who's been working heavily in Rust for the last year, I have to agree with you, here.

Look, there's a lot of folks who gripe about Rust; I used to be one of them. It's like someone took C-lang and pushed it to hard mode, but the core point keeps getting lost in these conversations: Rust never claimed to solve logic bugs, and nobody serious argues otherwise. What it does is remove an entire universe of memory-unsafety pitfalls that have historically caused catastrophic outages and security incidents.

The Cloudflare issue wasn’t about memory corruption or type confusion. It was a straight logic flaw. Rust can’t save you from that any more than Ada, Go, or Haskell can. Once you accept arbitrary external input, the compiler can’t enforce the invariants for you. You need validation, you need constraints, you need a spec, and you need tests that actually reflect the real world.

The idea that "only formally verified languages eliminate logic bugs" is technically correct but practically irrelevant for the scale Cloudflare operates at. Fully verified stacks exist, like seL4, but they are extremely expensive and restrictive. Production engineering teams are not going to rewrite everything in Coq. So we operate in the real world, where Rust buys us memory safety, better concurrency guarantees, and stricter APIs, but the humans still have to get the logic right.

This is not a Rust failure. It is the nature of software. If the industry switched from Rust to OCaml, Haskell, Ada, or C#, the exact same logic bug could still have shipped. Expecting Rust to prevent it misunderstands what problems Rust is designed to eliminate.

Rust does not stop you from writing the wrong code. It stops you from writing code that explodes in ways you did not intend. This wasn't the fault of the language, it was the fault of the folks who screwed up. You don't blame the hammer when you smack your thumb instead of a nail - you should blame your piss poor aim.

Re: Cloudflare outage should not have happened

#254

Post by b-man »

swiftcoder wrote: post #99
vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

> It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. We could also invest in tooling to make this kind of thing easier. Unclear why humans need to hand-normalise the database schema - isn't this exactly the kind of thing compilers are good at?

Normalization cannot be done by machines, because it depends on expressing the (and only the) predicate that corresponds to the business rule in question.

It requires apprehending the essence of the situation, something a machine cannot do.

Re: Cloudflare outage should not have happened

#255

Post by swiftcoder »

b-man wrote: post #254
swiftcoder wrote: post #99

Earlier quoted context omitted.

> It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. We could also invest in tooling to make this kind of thing easier. Unclear why humans need to hand-normalise the database schema - isn't this exactly the kind of thing compilers are good at?

Normalization cannot be done by machines, because it depends on expressing the (and only the) predicate that corresponds to the business rule in question. It requires apprehending the essence of the situation, something a machine cannot do.

Sure it can. Not in a vacuum, maybe, but with some guidance from the user as to dependency relations. Ideally you have enough data baked into your schema to infer those relations.

Re: Cloudflare outage should not have happened

#256

Post by kbdot »

kjgkjhfkjf wrote: post #213
mosura wrote: post #55

Earlier quoted context omitted.

Software people, especially coming through Rust, are falling into the old trap of believing if code is bug free it is reliable: it isn’t because there is a world of faults outside, including but not limited to the developer intentions. This inverts everything because structuring to be fault tolerant, of the right things, changes what is a good idea almost entirely.

To be fair to Rust, the issue was an "unwrap" in the Rust code[0]. "unwrap" means "if the operation did not succeed then panic". Production Rust code should not use "unwrap", and should instead have logic to handle the failure case. You don't need exotic formal verification methods to enforce this best practice. You just need a linter. [0] https://blog.cloudflare.com/18-november-2025-outage/#memory-...

I don’t know why anyone thinks that you can’t or shouldn’t use unwrap in a production environment. If this was enforced you’d likely end up with people using something like “.expect()” and there are real world cases where you KNOW that unwrapping will never cause issues.

The reality is the code should not have used unwrap, but that doesn’t mean using unwrap is bad.

Re: Cloudflare outage should not have happened

#257

Post by b-man »

swiftcoder wrote: post #255
b-man wrote: post #254

Earlier quoted context omitted.

Normalization cannot be done by machines, because it depends on expressing the (and only the) predicate that corresponds to the business rule in question. It requires apprehending the essence of the situation, something a machine cannot do.

Sure it can. Not in a vacuum, maybe, but with some guidance from the user as to dependency relations. Ideally you have enough data baked into your schema to infer those relations.

a fully normalized relation is one where the SQL (say) table in question represents one and only one predicate of your business rules.

It is literally impossible for that to be done automatically. Someone needs to look at the resulting code and confirm that that was the case.

Re: Cloudflare outage should not have happened

#258

Post by b-man »

vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

> That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms

I'm happy that we agree on the solution, but disagree only if it is cost worthy. About the cost, I took that into consideration when I wrote the conclusion:

> FAANG-style companies are unlikely to adopt formal methods or relational rigor wholesale. But for their most critical systems, they should. It’s the only way to make failures like this impossible by design, rather than just less likely.

There is an actionable plan in the article. It is possible to run teams like these. It is an economical decision of upper management to run the risk of having these outages vis-a-vis this alternative.

Re: Cloudflare outage should not have happened

#259

Post by b-man »

jacquesm wrote: post #26
dfabulich wrote: post #21

Earlier quoted context omitted.

That's entirely right. Products have to transition from fast-moving exploratory products to boring infrastructure. We have different goals and expectations for an ecommerce web app vs. a database, or a database vs. the software controlling an insulin pump. Having said that, at this point, Cloudflare's core DDOS-protection proxy should now be built more like an insulin pump than like a web app. This thing needs to nev…

Precisely. This is key infrastructure we're talking about not some kind of webshop.

Indeed. I was trying to make that point on my concluding paragraph.
Post reply on HN