Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding how Facebook disappeared from the internet

blog.cloudflare.com

Post reply on HN
Print view View original

11–20 of 280 posts

Re: Understanding how Facebook disappeared from the internet

#11

Post by mnordhoff »

"Because of this Cloudflare’s 1.1.1.1 DNS resolver could no longer respond to queries asking for the IP address of facebook.com or instagram.com."

The instagram.com zone itself uses a third-party DNS service and didn't go down. (But e.g. www.instagram.com is a CNAME to a zone on FB DNS.)

Re: Understanding how Facebook disappeared from the internet

#14

Post by yftsui »

mnordhoff wrote: post #11

"Because of this Cloudflare’s 1.1.1.1 DNS resolver could no longer respond to queries asking for the IP address of facebook.com or instagram.com." The instagram.com zone itself uses a third-party DNS service and didn't go down. (But e.g. www.instagram.com is a CNAME to a zone on FB DNS.)

That’s pretty much why during the downtime visit instagram.com showed a 503 from AWS instead.

Re: Understanding how Facebook disappeared from the internet

#16

Post by theshadowknows »

kube-system wrote: post #9
throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

Given my experience with DNS issues, I am guessing that they are running into dependencies along the way that assume/require DNS be available to function.

Yeah as part of my job I often have to work with our DNS team to provision say a subdomain or get some domain verified. They’ve got like…three people…trying to service thousands of teams across the enterprise. I do not envy their job at all.

Re: Understanding how Facebook disappeared from the internet

#17

Post by Wolfspirit »

nabaraz wrote: post #10

Is there a way to secure BGP? Some kind of BGP table backup and restore?

I think that it is possible to restore the previous state but the question is, if it makes sense. When do you decide that it was a failure? Facebook explicitly (even so automatically) told all others that they shouldn't use that routes anymore.

When it comes to Facebooks side I guess they do have backups of their BGP config. Applying them (probably remotely) however seems to be harder then expected when the whole infrastructure is down.

Re: Understanding how Facebook disappeared from the internet

#18

Post by powera »

phoe-krk wrote: post #4
andp97 wrote: post #2

No one. Cloudflare that write a report of some else network outage! AWESOW COMPANY!!!

Do you really think that Facebook is going to write one right now? It's good that we have some coverage from companies that have some stake in the game because they are also affected by the outage, even if only partially.

Facebook has to make some public statement; the shareholders will demand it.

I expect the detail level to be roughly "an automated system pushed a broken configuration"; that is to say, there probably won't be any interesting information at all for the Hacker News crowd.

I doubt that this was caused by "hackers" or "hostile governments" or "dissident employees upset about Facebook privacy issues", and also doubt that Facebook would admit such if it were true unless they were legally required to do so.

Post reply on HN