Cloudflare outage should not have happened • Hacker News Board

Re: Cloudflare outage should not have happened

#51

Post by robocat » Wed, Nov 26, 2025, 5:49 PM UTC

jacquesm wrote: post #37

frumplestlatz wrote: post #32

Earlier quoted context omitted.

I work in formal verification at a FAANG. It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths, and even then, the properties asserted by formal verification are often quite a bit less powerful than necessary to truly guarantee something useful. I want formal verification everywhere. I believe in provable correctness. I…

Ok, let's start off with holding them to the same standards as avionics software development. The formal verification can wait.

Anyone in avionics software dev to give an opinion?

I would presume there's the same issue as parent said:

  It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths

Re: Cloudflare outage should not have happened

#52

Post by this_user » Wed, Nov 26, 2025, 5:50 PM UTC

Of course it shouldn't have happened. But if you run infrastructure as complex as this on the scale that they do, and with the agility that they need, then it was bound to happen eventually. No matter how good you are, there is always some extremely unlikely chain of events that will lead to a catastrophic out. Given enough time, that chain will eventually happen.

Re: Cloudflare outage should not have happened

#53

Post by wathef » Wed, Nov 26, 2025, 5:51 PM UTC

simlevesque wrote: post #38

jacquesm wrote: post #26

Earlier quoted context omitted.

Precisely. This is key infrastructure we're talking about not some kind of webshop.

Yeah but the anti-DDOS feature needs to react to new methods all the time, it's not a static thing you build once and it works forever. An insulin pump is very different. Your human body, insulin, and physics aren't changing any time soon.

an insulin pump is a good metaphor; insulin as a hormone has a lot of interactions and the pump itself, if not wanting to unalive its user, should (most do not) account for external variables, such as: exercise, heart rate, sickness, etc. these variables are left for the user to deal with, and in this case, is a subpar experience in managing a condition.

Re: Cloudflare outage should not have happened

#54

Post by ottah » Wed, Nov 26, 2025, 5:52 PM UTC

frumplestlatz wrote: post #32

jacquesm wrote: post #25

Earlier quoted context omitted.

When you're powering this large a fraction of the internet is it even an option not to work like that? You'd think that with that kind of market cap resource constraints should no longer be holding you back from doing things properly.

I work in formal verification at a FAANG. It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths, and even then, the properties asserted by formal verification are often quite a bit less powerful than necessary to truly guarantee something useful. I want formal verification everywhere. I believe in provable correctness. I…

I would argue the largest CDN provider in the world is a critical path.

Re: Cloudflare outage should not have happened

#55

Post by mosura » Wed, Nov 26, 2025, 5:52 PM UTC

vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

Software people, especially coming through Rust, are falling into the old trap of believing if code is bug free it is reliable: it isn’t because there is a world of faults outside, including but not limited to the developer intentions.

This inverts everything because structuring to be fault tolerant, of the right things, changes what is a good idea almost entirely.

Re: Cloudflare outage should not have happened

#56

Post by bambax » Wed, Nov 26, 2025, 5:53 PM UTC

vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

But does "formally verified code" really go in the same bag as "normalized database" and ensuring data integrity at the database level? The former is immensely complex and difficult; the other two are more like sound engineering principles?

Re: Cloudflare outage should not have happened

#57

Post by dpark » Wed, Nov 26, 2025, 5:54 PM UTC

jacquesm wrote: post #37

frumplestlatz wrote: post #32

Earlier quoted context omitted.

I work in formal verification at a FAANG. It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths, and even then, the properties asserted by formal verification are often quite a bit less powerful than necessary to truly guarantee something useful. I want formal verification everywhere. I believe in provable correctness. I…

Ok, let's start off with holding them to the same standards as avionics software development. The formal verification can wait.

I don’t understand why anyone should want this. Why should normal engineering efforts be held to the same standards as life-critical systems? Why would anyone expect that CloudFlare DDoS protection be built to the standards of avionics equipment?

Also if we’re being fair, avionics software is far narrower in scope than just “software in general”. And even with that Boeing managed to kill a bunch of people with shitty software.

Re: Cloudflare outage should not have happened

#58

Post by luhn » Wed, Nov 26, 2025, 5:54 PM UTC

cyberax wrote: post #30

Analemma_ wrote: post #13

Earlier quoted context omitted.

When I was at AWS, when we did postmortems on incidents we called it "root cause analysis", but it was understood by everyone that most incidents are multicausal and the actual analyses always ended up being fishbone diagrams. Probably there are some teams which don't do this and really do treat RCA as trying to find a sole root cause, but I think a lot of "getting mad at RCA" is bikeshedding the terminology, and not…

> to find a sole root cause "Six billion years ago the dust around the young Sun coalesced into planets"

"Workaround: If we wait long enough, the earth will eventually be consumed by the sun."

https://xkcd.com/1822/

Re: Cloudflare outage should not have happened

#59

Post by jacquesm » Wed, Nov 26, 2025, 5:55 PM UTC

robocat wrote: post #51

jacquesm wrote: post #37

Earlier quoted context omitted.

Ok, let's start off with holding them to the same standards as avionics software development. The formal verification can wait.

Anyone in avionics software dev to give an opinion? I would presume there's the same issue as parent said: It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths

> Anyone in avionics software dev to give an opinion?

I've done some for fuel estimation of freighter jets (not quite avionics but close enough to get a sense for the development processes) and the amount of rigor involved in that one project made me a better developer for the rest of my career. Was it slow? Yes, it was very slow. A couple of thousand lines of code, a multiple of that in tests for a very long time compare to what it would normally take me.

But within the full envelope of possible inputs it performed exactly as advertised. The funny thing is that I'm not particularly proud of it, it was the process that kept things running even when my former games programmer mentality would have long ago said 'ship it'.

Some things you just need to do properly, or not at all.

Re: Cloudflare outage should not have happened

#60

Post by tracker1 » Wed, Nov 26, 2025, 5:56 PM UTC

echelon wrote: post #46

locknitpicker wrote: post #4

This sort of Monday morning quarterbacking is pointless and only serves as a way for random bloggers to try to grab credit without actually doing or creating any value.

Like your comment? j/k :) I'm using this incident to draw attention to Rust's panic behavior. Rust could use additional language features to help us write mostly panic-free* code and statically catch even transitive dependencies that might subject us to unnecessary panics. We've been talking about it on our team and to other Rust folks, and I think it's worth building a proposal around. Rust should have a way to stat…

It's already in the box... there's a bunch of options from unwrap_or, etc... to actually checking the error result and dealing with it cleanly... that's not what happened.

Not to mention the possibility of just bumping up through Result chaining with an app specific error model. The author chose neither... likely because they want the app to crash/reload from an external service. This is often the best approach to an indeterminate or unusable state/configuration.