Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage should not have happened

ebellani.github.io

Post reply on HN
Print view View original

261–265 of 265 posts

Re: Cloudflare outage should not have happened

#261

Post by swiftcoder »

b-man wrote: post #257
swiftcoder wrote: post #255

Earlier quoted context omitted.

Sure it can. Not in a vacuum, maybe, but with some guidance from the user as to dependency relations. Ideally you have enough data baked into your schema to infer those relations.

a fully normalized relation is one where the SQL (say) table in question represents one and only one predicate of your business rules. It is literally impossible for that to be done automatically. Someone needs to look at the resulting code and confirm that that was the case.

Or they need to write the business rules down, in some sort of expressive format that can be tested against the requirements... Say, an artificial language of some sort. One might call it "programming"

Re: Cloudflare outage should not have happened

#262

Post by yodon »

packetslave wrote: post #166
yodon wrote: post #17

Earlier quoted context omitted.

>Gradual deployments are a more reliable defense against bugs than careful programming The challenge, as I understand it, is that the feature in question had an explicit requirement of fast, wide deployment because of the need to react in real time to changing external attacker behaviors.

yep, and it was this exact requirement that also caused the exact same outage back in 2013 or so. DDoS rules were pushed to the GFE (edge proxy) every 15 seconds, and a bad release got out. Every single GFE worldwide crashed within 15 seconds. That outage is in the SRE book.

Is there a link to the SRE book?

Re: Cloudflare outage should not have happened

#263

Post by fragmede »

yodon wrote: post #262
packetslave wrote: post #166

Earlier quoted context omitted.

yep, and it was this exact requirement that also caused the exact same outage back in 2013 or so. DDoS rules were pushed to the GFE (edge proxy) every 15 seconds, and a bad release got out. Every single GFE worldwide crashed within 15 seconds. That outage is in the SRE book.

Is there a link to the SRE book?

https://sre.google/sre-book/table-of-contents/

Re: Cloudflare outage should not have happened

#264

Post by b-man »

yearolinuxdsktp wrote: post #92
vessenes wrote: post #5

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…

Not to mention that perfectly normalizing a database always incurs join overhead that limits horizontal scalability. In fact, denormalization is required to achieve scale (with a trade-off). I’m not sure how formal verification would’ve prevented this issue from happening. In my experience, it’s unusual to have to specify a database name in the query. How could have formal verification covered this outcome? The recom…

> Not to mention that perfectly normalizing a database always incurs join overhead that limits horizontal scalability. In fact, denormalization is required to achieve scale (with a trade-off).

This is just not true, at least not in general. Inserting on a normalized design is usually faster, due to smaller index sizes, fewer indexes and fitting more rows per page.

Re: Cloudflare outage should not have happened

#265

Post by gavinhoward »

frumplestlatz wrote: post #32
jacquesm wrote: post #25

Earlier quoted context omitted.

When you're powering this large a fraction of the internet is it even an option not to work like that? You'd think that with that kind of market cap resource constraints should no longer be holding you back from doing things properly.

I work in formal verification at a FAANG. It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths, and even then, the properties asserted by formal verification are often quite a bit less powerful than necessary to truly guarantee something useful. I want formal verification everywhere. I believe in provable correctness. I…

Hey, where can I apply to a job like yours? I may not be smart enough, but I may be. And I am very interested in formal verification.
Post reply on HN