Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

Post reply on HN
Print view View original

31–40 of 265 posts

Re: Cloudflare outage should not have happened

#31

Post by dzikimarian »

SoftTalker wrote: post #15
vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

Why is being able to "capture the market" something we want to encourage? This leads to monopolies or oligopolies and makes possible various types of abuse that a free competitive market would normally correct. If you're going to step into the role of managing a large percentage of public internet traffic, maybe you need to be held to a different standard and set of rules than a startup trying to get a foothold among…

You would have to completely flip how funding works. As of now most VCs have abysmal returns, so heightening the bar is last thing on their mind.

Re: Cloudflare outage should not have happened

#32

Post by frumplestlatz »

jacquesm wrote: post #25
vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

When you're powering this large a fraction of the internet is it even an option not to work like that? You'd think that with that kind of market cap resource constraints should no longer be holding you back from doing things properly.

I work in formal verification at a FAANG.

It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths, and even then, the properties asserted by formal verification are often quite a bit less powerful than necessary to truly guarantee something useful.

I want formal verification everywhere. I believe in provable correctness. I wish we could hire people capable of always writing software to that standard and maintaining those proofs alongside their work.

We really can’t, though. Its a frustrating reality of being human — we know how to do it better, but nearly all of even the smartest engineers we can hire are not smart enough.

Re: Cloudflare outage should not have happened

#33

Post by jacquesm »

bell-cot wrote: post #16
mikece wrote: post #2

Yes, pretty basic looking mistakes that, from the outside, make many wonder how this got through. Though analyzing the post-mortem makes me think of the MV Dali crashing into the Francis Scott Key bridge in Baltimore: the whole thing started with a single loose wire which set off a cascading failure. CF's situation was similar in a few ways though finding a bad query (and .unwrap() in production code rather than test…

> ...makes me think of the MV Dali crashing... Yes. Though compared to Cloudflare's infrastructure, the Dali is a wooden rowboat. And CF doesn't have the "...or people will die" safety criticality.

> And CF doesn't have the "...or people will die" safety criticality.

I disagree with that. Just because you can't point to people falling off a bridge into the water doesn't mean that outages of the web at this scale will not lead to fatalities.

Re: Cloudflare outage should not have happened

#34

Post by nyrikki »

Hindsight bias is always easier but:

> FAANG-style companies are unlikely to adopt formal methods or relational rigor wholesale. But for their most critical systems, they should. It’s the only way to make failures like this impossible by design, rather than just less likely.

That relational rigor imposes what one chooses to be true, it isn’t a universal truth.

The frame problem and the qualification problem apply here.

The open domain frame problem == HALT.

When you can for a problem into the relational model things are nice but not everything can be reduced to a trivial property.

That is why Codd had to as nulls etc..

You can choose to decide that the queen is rich OR pigs can fly; but a poor queen doesn’t result in flying pigs.

Choice over finite sets == finite indexes over sets == PEM

If you can restrict your problems to where the Entscheidungsproblem is solvable you can gain many benefits

But it is horses for courses and sub TC.

Re: Cloudflare outage should not have happened

#35

Post by PunchyHamster »

> but it clearly needs a distinct and a limit, since these seem to be crucial business rules.

Isn't that just... wrong ? Throwing arbitrary limit (vs maybe having some alert when the table is too long) would just silently truncate the list

Anybody can be backseat engineer by throwing out industry's best practices like they were gospel but you have to look at entire system, not just the database part

Re: Cloudflare outage should not have happened

#37

Post by jacquesm »

frumplestlatz wrote: post #32
jacquesm wrote: post #25

Earlier quoted context omitted.

When you're powering this large a fraction of the internet is it even an option not to work like that? You'd think that with that kind of market cap resource constraints should no longer be holding you back from doing things properly.

I work in formal verification at a FAANG. It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths, and even then, the properties asserted by formal verification are often quite a bit less powerful than necessary to truly guarantee something useful. I want formal verification everywhere. I believe in provable correctness. I…

Ok, let's start off with holding them to the same standards as avionics software development. The formal verification can wait.

Re: Cloudflare outage should not have happened

#38

Post by simlevesque »

jacquesm wrote: post #26
dfabulich wrote: post #21

Earlier quoted context omitted.

That's entirely right. Products have to transition from fast-moving exploratory products to boring infrastructure. We have different goals and expectations for an ecommerce web app vs. a database, or a database vs. the software controlling an insulin pump. Having said that, at this point, Cloudflare's core DDOS-protection proxy should now be built more like an insulin pump than like a web app. This thing needs to nev…

Precisely. This is key infrastructure we're talking about not some kind of webshop.

Yeah but the anti-DDOS feature needs to react to new methods all the time, it's not a static thing you build once and it works forever.

An insulin pump is very different. Your human body, insulin, and physics aren't changing any time soon.

Re: Cloudflare outage should not have happened

#39

Post by PunchyHamster »

vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

I'd not be surprised if root of the issue was some engineer who didn't add DB selector because in other SQL engines SELECT like that would select from currently connected database vs all of them

Re: Cloudflare outage should not have happened

#40

Post by lysace »

kjuulh wrote: post #11

It did happen, and cloudflare should learn from it, but not just the technical reasons. Instead of focusing on the technical reasons why, they should answer how such a change bubbled out to cause such a massive impact instead. Why: Proxy fails requests Why: Handlers crashed because of OOM Why: Clickhouse returns too much data Why: A change was introduced causing double the amount of data Why: A central change was rol…

Very quick rollout is crucial for this kind of service. On top of what you wrote, institutionalizing rollback by default if something catastrophically breaks should be the norm.

Been there in those calls, begging to people in charge who perhaps shouldn't have been, "eh, maybe we should attempt a rollback to the last known good state? cause, it, you know.... worked". But investigating further before making any change always seems to be the preferred action to these people. Can't be faulted for being cautious and doing things properly, right? I kid you not - this is their instinct.

If I recall correctly it took CF 2 hours to roll back the broken changes.

So if I were in charge of Cloudflare (4-5k employees) I'd both look at the processes and the people in charge.

Post reply on HN