Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update about the October 4th outage

engineering.fb.com

Post reply on HN
Print view View original

231–239 of 239 posts

Re: Update about the October 4th outage

#231

Post by dTal »

EricE wrote: post #220
dTal wrote: post #165

Earlier quoted context omitted.

The app is designed under the assumption that Facebook servers are never down. If you can't reach the servers, the problem is assumed to be client-side, in which case they have decided the best UI is to keep retrying (not unreasonably in a mobile context). The only way to disambiguate "no internet service" (extremely common) with "Facebook dropped off the internet" (black-swan rare) is to ping some other, third party…

>The app is designed under the assumption that Facebook servers are never down. Which was and is a lame assumption. Stuff happens. SMTP wouldn't even be phased by this; it would just pick up where it left off. I've seen far too many applications fail in bizzare ways because people make unrealistic assumptions like "X will ALWAYS be there". Sure it's highly unlikely, but when you have multiple things making the same d…

How would you address the issue?

Re: Update about the October 4th outage

#232

Post by Dylan16807 »

manquer wrote: post #203
Dylan16807 wrote: post #159

Earlier quoted context omitted.

> You are effectively asking is why is there a single routing table for the internet. No, they're asking why a single set of routers is in charge of announcing BGP routes for all of facebook. If you have multiple ASes, with independent configuration sources and independent routers broadcasting them, it's a lot harder to break everything at once.

Facebook definitely has multiple ASes and likely rolling BGP update strategies already. This isn't the first high profile BGP screwup after all. However if even one set of routers were misconfigured and it was announcing incorrect routes for all their ASes as result of the issue then their peers will not typically drop that set alone automatically. BGP doesn't have a Paxos / Raft style smart consensus algorithms, it…

The idea is that there would be multiple layers preventing one set of routers from announcing routes for "all their ASes", correct or not.

The config database would reject routes for the wrong ASes. The router would reject it. The peers would be told "add filters so you only accept these ASes from these routers".

Maybe they have all that and it somehow broke anyway? But what it looks like from the outside is that all the ASes are controlled by the same system.

> their peers will not typically drop that set alone automatically.

I'm not sure what this sentence means.

Re: Update about the October 4th outage

#234

Post by Thorrez »

switch007 wrote: post #92
Thorrez wrote: post #69

Earlier quoted context omitted.

A New York Times reporter said employees had trouble getting in, and I don't see a retraction. So I believe that is true: https://twitter.com/sheeraf/status/1445099150316503057

Do we hold journalists posting on Twitter to the same standards as writing in newspapers and their websites? Genuine question

No, but I hold journalists posting on Twitter to a higher standard than someone unknown posting on Twitter. I don't see why she would lie.

Re: Update about the October 4th outage

#235

Post by implements »

colechristensen wrote: post #224
implements wrote: post #137

Earlier quoted context omitted.

I briefly worked on access control a long time ago - aren’t they supposed to cache credentials locally so that individual doors still function autonomously during a fire / loss of power and comms?

Eh I would assume that would only be necessary if the lock mechanism kept people in, the only place I’ve been that did that was a defense contractor, and even then there were alarmed emergency doors. Everywhere else had either no exit control or trivially bypassed exit control.

Wasn’t defence, but somewhere with highly competitive commercially sensitive engineered components. You needed a card to open a door in either direction, but exit routes all had “Break to Exit” bypass switches - which lazy engineers often did, particularly if the door wasn’t covered by CCTV.

My curiously downvoted point was, I’m surprised an internet problem would stop an high security access control system from functioning - they’re supposed to be designed to cope with that and continue to work autonomously in emergencies.

Re: Update about the October 4th outage

#236

Post by oarsinsync »

TheCoelacanth wrote: post #229
oarsinsync wrote: post #204

Earlier quoted context omitted.

> Some applications may have seen increased load and suffered due to server resourcing constraints, caused by applications like the above failing to fail gracefully, and instead polling more aggressively. I had Cloudflare's woes in mind when I wrote that.

So your point is that it didn't slow down the entire Internet, only the parts of the Internet that use DNS (damn near all of them)?

Based on cloudflare's own blog[0], 1% of queries went unanswered for about 10 mins, and were significantly delayed for a few hours. 5% of queries were delayed for a couple of hours. 95% of queries continued to be resolved as normal.

This is referring explicitly to users of 1.1.1.1, which is likely not the same infrastructure as domains hosted by cloudflare dns.

[0] https://blog.cloudflare.com/october-2021-facebook-outage/

Re: Update about the October 4th outage

#238

Post by manquer »

Dylan16807 wrote: post #232
manquer wrote: post #203

Earlier quoted context omitted.

Facebook definitely has multiple ASes and likely rolling BGP update strategies already. This isn't the first high profile BGP screwup after all. However if even one set of routers were misconfigured and it was announcing incorrect routes for all their ASes as result of the issue then their peers will not typically drop that set alone automatically. BGP doesn't have a Paxos / Raft style smart consensus algorithms, it…

The idea is that there would be multiple layers preventing one set of routers from announcing routes for "all their ASes", correct or not. The config database would reject routes for the wrong ASes. The router would reject it. The peers would be told "add filters so you only accept these ASes from these routers". Maybe they have all that and it somehow broke anyway? But what it looks like from the outside is that all…

> config database would reject

> router would reject it.

Any of these hardware could have bugs, if one of them announces wrongly it will be propagated wrongly by all other ASes peering with them and to the next level so on and on. That is the point, at this level it is possible to fuck up globally.

> peers would be told "add filters so you only accept these ASes from these routers".

There are hundreds of ISPs , all of them peer cannot directly with each other. Routes are propagated downstream and upstream it is a web of networks running on trust.

While filtering is built into most implementations ( sadly not the protocol itself), practically ISPs have no easy way to determine which AS can actually originate which other AS'es traffic, so they don't actually implement a lot of filtering. Remember traffic can have more than 2 hops. Effectively that means you would be routing traffic for AS 3/4 hops away. Neither you nor your peer would know anything about it or whether you can trust it etc.

Even if some ISPs do drop/block the announcements, unless every single AS also implements the block there won't be an impact. Traffic would route through ASes which don't have filtering and announce the routes incorrectly . For example say AT&T blocks an incorrectly announced FB route, but British Telecom does not, BGP is designed to assume that FB has lost peering with AT&T and route all traffic for FB via British telecom.

If filtering was robustly possible we wouldn't have periodic BGP hijacking incidents as we do whether accidental or maliciously. The famous Pakistan Telecom Youtube hijacking [2] or as recently as April-2021 [3] or incidents over the last few years usually authoritative governments (such as China/Russia etc) but with impact well beyond their networks.

[1] http://www.bgpexpert.com/article.php?article=145

[2] https://www.ripe.net/publications/news/industry-developments...

[3] https://blog.apnic.net/2021/04/26/a-major-bgp-route-leak-by-...

Post reply on HN