Understanding how Facebook disappeared from the internet • Hacker News Board

Re: Understanding how Facebook disappeared from the internet

#221

Post by belorn » Tue, Oct 05, 2021, 9:09 AM UTC

throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

It just a guess, but from experience with BGP and associated redundancy systems that most likely was in place, if everything doesn't return immediately then you have a big fight on your hand to not only stop the non-functional redundancy but also reestablish the peer connections with associted hearthbeat/processes for establishing and maintaining the peer connection. My understanding from what people write about configuring BGP and the system around it seems to imply that the best practice in this circumstance is to kill everything, fix the original error and then turn on things slowly again. Then fix the broken redundancy configuration. Then test the redundancy system regularly in the future.

Re: Understanding how Facebook disappeared from the internet

#222

Post by pitkali » Tue, Oct 05, 2021, 9:10 AM UTC

phtrivier wrote: post #216

ccakes wrote: post #211

Earlier quoted context omitted.

You need to call Facebook to find out what your friends have been up to but their number has temporarily disappeared from the phone book.

That would have been accurate for a DNS outage ; but with my layman understanding of BGP, I would say the analogy would be something between "...but their phone line is broken" and "...but they disappeared from the phone book because they don't have a phone line any more" . Is that right ?

Is that an interesting distinction for the target audience, though?

Re: Understanding how Facebook disappeared from the internet

#223

Post by Hikikomori » Tue, Oct 05, 2021, 9:21 AM UTC

p4bl0 wrote: post #212

Hikikomori wrote: post #66

Earlier quoted context omitted.

Once they start adversiting again it should only take a few minutes at most for most ISPs to get it.

But then it is the DNS that had to propagate from the now accessible authoritative servers.

Shouldn't take long once they are up and responding again?

Re: Understanding how Facebook disappeared from the internet

#224

Post by nroets » Tue, Oct 05, 2021, 9:29 AM UTC

PaulRobinson wrote: post #208

It's been 20+ years since I had reason to know the internals of BGP, but I still carry a slight nervousness whenever I think about it. In many ways, it's beautiful in that it allows Autonomous Systems to come together and independently make the Internet. However, I always felt it was brittle in that it relied every AS to have edge routers aware of the entire AS routing map, and so when those routers went down hard, w…

So much of the internet is about translating addresses from one format to another:

Human readable to IP: DNS

Private IP to Public IP: NAT

MAC to Private IP: ARP

Public IP to AS: BGP

With a block chain DNS it may be possible to replace all of them with a single protocol, but it may well turn out to be an academic exercise.

Re: Understanding how Facebook disappeared from the internet

#225

Post by jgrahamc » Tue, Oct 05, 2021, 9:33 AM UTC

alksjdalkj wrote: post #49

Question about the WARP map - I assume the grey countries are places where Cloudflare doesn't have any presence, but what about Egypt/Oman? Why are they green? And why is Australia orange and not red?

We didn't have enough data from those locations.

Re: Understanding how Facebook disappeared from the internet

#226

Post by jgrahamc » Tue, Oct 05, 2021, 9:33 AM UTC

davidjytang wrote: post #190

alksjdalkj wrote: post #49

Question about the WARP map - I assume the grey countries are places where Cloudflare doesn't have any presence, but what about Egypt/Oman? Why are they green? And why is Australia orange and not red?

Also curious about the green part.

It's an anomaly. I think that was right on the edge of "not enough traffic to report on".

Re: Understanding how Facebook disappeared from the internet

#227

Post by jgrahamc » Tue, Oct 05, 2021, 9:34 AM UTC

mnordhoff wrote: post #11

"Because of this Cloudflare’s 1.1.1.1 DNS resolver could no longer respond to queries asking for the IP address of facebook.com or instagram.com." The instagram.com zone itself uses a third-party DNS service and didn't go down. (But e.g. www.instagram.com is a CNAME to a zone on FB DNS.)

Thanks. I'll correct that.

Re: Understanding how Facebook disappeared from the internet

#228

Post by jgrahamc » Tue, Oct 05, 2021, 9:37 AM UTC

hinkley wrote: post #100

Ansil849 wrote: post #72

Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it. Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing gl…

They've done this in previous outages and people didn't like it then, either. Generally big companies don't talk smack about other big companies having internet-wide issues, unless those issues are directly caused by the other company. For instance, when Google talked about Cloudbleed, which was when Cloudflare vomited millions of secrets all over Google's caching heirarchy and Google had to manually clean it up. I t…

which is that Cloudflare is the last company that should be criticizing everyone else

There is no criticism of Facebook in our blog post.

Re: Understanding how Facebook disappeared from the internet

#229

Post by [deleted] » Tue, Oct 05, 2021, 9:48 AM UTC

[deleted]

Re: Understanding how Facebook disappeared from the internet

#230

Post by grlass » Tue, Oct 05, 2021, 9:51 AM UTC

nroets wrote: post #224

PaulRobinson wrote: post #208

It's been 20+ years since I had reason to know the internals of BGP, but I still carry a slight nervousness whenever I think about it. In many ways, it's beautiful in that it allows Autonomous Systems to come together and independently make the Internet. However, I always felt it was brittle in that it relied every AS to have edge routers aware of the entire AS routing map, and so when those routers went down hard, w…

So much of the internet is about translating addresses from one format to another: Human readable to IP: DNS Private IP to Public IP: NAT MAC to Private IP: ARP Public IP to AS: BGP With a block chain DNS it may be possible to replace all of them with a single protocol, but it may well turn out to be an academic exercise.

Namecoin [1] was an early fork of bitcoin, that did blockchain based DNS, with the ideas being actively discussed by Satoshi Nakamoto [2].

That deals with DNS (though how much has it been used?), but what about the layers of the stack?

[1] https://en.wikipedia.org/wiki/Namecoin

[2] https://bitcointalk.org/index.php?topic=1790.222