Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

Post reply on HN
Print view View original

171–180 of 265 posts

Re: Cloudflare outage should not have happened

#171

Post by gishh »

ViewTrick1002 wrote: post #103
mosura wrote: post #55

Earlier quoted context omitted.

Software people, especially coming through Rust, are falling into the old trap of believing if code is bug free it is reliable: it isn’t because there is a world of faults outside, including but not limited to the developer intentions. This inverts everything because structuring to be fault tolerant, of the right things, changes what is a good idea almost entirely.

Rust generally forces you to acknowledge these faults. The problem is managing them in a sane way, which for Rust in many cases simply is failing loudly. Compared to than many other languages which preferring chugging along and hoping that no downstream corruption happens.

[flagged]

Re: Cloudflare outage should not have happened

#172

Post by vablings »

lenkite wrote: post #126
frumplestlatz wrote: post #47

Earlier quoted context omitted.

Agreed. I left out any commentary on `.unwrap()` from my original comment, but it’s an obvious example of something that should never have appeared in critical code.

ON HN, just a couple of years ago, a famous Rust programmer said that it is OK to use unwrap. Rustaceans supported this position. Cloudflare merely followed the community standard. Using unwrap() in Rust is Okay https://news.ycombinator.com/item?id=32385102 https://burntsushi.net/unwrap/

Also, in this use case catching the panic and completely forgetting that the function was ever called in the first place is completely acceptable. In web frameworks such as Dioxus/Axum if your users request causes a panic it does not bring down the whole web server it just invalidates that specific request

Re: Cloudflare outage should not have happened

#173

Post by knorker »

No, this is nonsense and look like university student naivety.

What caused it was rolling out a change and moving on to the next recipient without checking if the previous task instantly died.

You can't prevent all crash bugs, but you can check if you are lasering your whole prod.

Re: Cloudflare outage should not have happened

#174

Post by vablings »

lysace wrote: post #40
kjuulh wrote: post #11

It did happen, and cloudflare should learn from it, but not just the technical reasons. Instead of focusing on the technical reasons why, they should answer how such a change bubbled out to cause such a massive impact instead. Why: Proxy fails requests Why: Handlers crashed because of OOM Why: Clickhouse returns too much data Why: A change was introduced causing double the amount of data Why: A central change was rol…

Very quick rollout is crucial for this kind of service. On top of what you wrote, institutionalizing rollback by default if something catastrophically breaks should be the norm. Been there in those calls, begging to people in charge who perhaps shouldn't have been, "eh, maybe we should attempt a rollback to the last known good state? cause, it, you know.... worked". But investigating further before making any change…

It does seem insane to me that there isnt a process to catch the panic, unwind back to a reasonable place in the call stack, load the last known good configuration and continue execution as normal. You would go from having a global 2 hour outage to a warning on a dashboard that can be investigated in a timely manner rather than blowing up half the internet

Re: Cloudflare outage should not have happened

#175

Post by notepad0x90 »

The real RCA (IMHO) is not simulating outages in production as part of reliability engineering.

Whatever process was stuck in a loop, crashed, or whatever service (db, dns,etc..) was unavailable, that outage scenario can be simulated. Changes can have an automated rollback requirement.

My take away is that CF has single points of failure they're aware of, and for business reasons, they've decided to not have a redundancy/failover.

> ...and formally verified code, this bug would not have happened.

That's what I mean, "we should have caught the bug" , yeah, but that isn't reliability engineering. You assume there will be bugs/outages and prepare for them instead. What happens if the entire DB entered a weird state and was spitting out valid results with incorrect values? What happens if it accepts connections and just stalls?

You prepare for bugs that don't yet exist, you fix bugs that do exist.

Re: Cloudflare outage should not have happened

#176

Post by vel0city »

robocat wrote: post #51
jacquesm wrote: post #37

Earlier quoted context omitted.

Ok, let's start off with holding them to the same standards as avionics software development. The formal verification can wait.

Anyone in avionics software dev to give an opinion? I would presume there's the same issue as parent said: It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths

Compare the prices for type certified parts on aircraft compared to comparable (but not proven) similar parts in the automotive space. Its crazy how much more expensive actually proving these parts perform to spec to the level required by aviation law.

It wouldn't surprise me to find doing the same kind of certifications for complex avionics software to be the same.

Re: Cloudflare outage should not have happened

#178

Post by avereveard »

"If only the world was perfect the world would be perfect"

Author fails to mention how to actually formally verify this asynchronous globally replicated product. He may have solved the delivery theorem and if that's so I encourage him sharing the results.

> No nullable fiels.

Author appears to have not formally verified his post's grammar.

Re: Cloudflare outage should not have happened

#179

Post by Dylan16807 »

vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

Formally verifying code is an enormous endeavor.

But a normalized database without NULL should not be a significant burden.

Re: Cloudflare outage should not have happened

#180

Post by saati »

gishh wrote: post #171
ViewTrick1002 wrote: post #103

Earlier quoted context omitted.

Rust generally forces you to acknowledge these faults. The problem is managing them in a sane way, which for Rust in many cases simply is failing loudly. Compared to than many other languages which preferring chugging along and hoping that no downstream corruption happens.

[flagged]

Was it a memory error or a data race? No. Rust only promises that those won't happen in safe Rust. What is embarrassing is trying to pin this on a specific programming language.
Post reply on HN