Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding how Facebook disappeared from the internet

blog.cloudflare.com

Post reply on HN
Print view View original

71–80 of 280 posts

Re: Understanding how Facebook disappeared from the internet

#71

Post by cyanydeez »

Hikikomori wrote: post #48
throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

Restoring is just as simple as flipping the switch again, but access to that switch is another matter when your internal network is also down and you cannot even get access to your office or datacenters.

engineers will tell you that not everything is reversible even if theres no specific cqpacity issue

Re: Understanding how Facebook disappeared from the internet

#72

Post by Ansil849 »

Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it.

Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing glorified bikeshedding, and providing an explainer of how BGP works that can be gleaned by reading virtually any introductory text on it, and using it as a marketing opportunity.

Re: Understanding how Facebook disappeared from the internet

#73

Post by termau »

throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

I feel like it just confuses the issue with a bunch of unnecessary babble about DNS, theres better ways to read about how BGP works without confusing a bunch of different things. The only part of the article that was relevant was 'Routes were withdrawn' - the rest being a consequence of that.

Re: Understanding how Facebook disappeared from the internet

#74

Post by foobarian »

Hikikomori wrote: post #48
throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

Restoring is just as simple as flipping the switch again, but access to that switch is another matter when your internal network is also down and you cannot even get access to your office or datacenters.

What I don't get is why there is no dead-man switch mechanism in place to roll back the configuration automatically unless someone confirms it positively. Kind of how screen resolution rolls back if you don't ack it. I used to always run a "(sleep 600; iptables -F) &" when messing with remote personal stuff just in case I lock myself out.

I suppose with something like BGP it would be very difficult to get such a fallback working given how distributed the system is, and even more difficult to keep it exercised and tested.

Re: Understanding how Facebook disappeared from the internet

#75

Post by cyanydeez »

toomuchtodo wrote: post #39
cwilkes wrote: post #38

Updating BGP configs should go through a flowchart like this: Do you want to update BGP? No: exit Yes: type this random 100 character phrase to continue no copy paste

“Are you sure? Can people you need to recover from this change already in the building?”

not just in the building, but are there any microsoft enabled security doors between you and them

Re: Understanding how Facebook disappeared from the internet

#76

Post by cyanydeez »

Wolfspirit wrote: post #17
nabaraz wrote: post #10

Is there a way to secure BGP? Some kind of BGP table backup and restore?

I think that it is possible to restore the previous state but the question is, if it makes sense. When do you decide that it was a failure? Facebook explicitly (even so automatically) told all others that they shouldn't use that routes anymore. When it comes to Facebooks side I guess they do have backups of their BGP config. Applying them (probably remotely) however seems to be harder then expected when the whole inf…

reading krebs suggest they couldnt do anything because all the security ran through the same computers keeping facebook.com

they basically ran into a footgun

Re: Understanding how Facebook disappeared from the internet

#77

Post by missedthecue »

Ansil849 wrote: post #72

Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it. Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing gl…

I mean... they're in the business of keeping websites online. It's natural that they document these events both for their own research and to market their product.

Re: Understanding how Facebook disappeared from the internet

#79

Post by rjvir »

Ansil849 wrote: post #72

Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it. Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing gl…

It's merely an informative blog post on a topic many people are interested in. I see nothing wrong with Cloudflare just explaining what happened, even though they had nothing to do with it.

Re: Understanding how Facebook disappeared from the internet

#80

Post by cyanydeez »

powera wrote: post #18
phoe-krk wrote: post #4

Earlier quoted context omitted.

Do you really think that Facebook is going to write one right now? It's good that we have some coverage from companies that have some stake in the game because they are also affected by the outage, even if only partially.

Facebook has to make some public statement; the shareholders will demand it. I expect the detail level to be roughly "an automated system pushed a broken configuration"; that is to say, there probably won't be any interesting information at all for the Hacker News crowd. I doubt that this was caused by "hackers" or "hostile governments" or "dissident employees upset about Facebook privacy issues", and also doubt that…

now adays, sharehooders maybe be just as quiet as everyone at facebook knowing making a big deal could lose them value, and theyd rather sell.

shareholders aint much these days of cancerous growth

Post reply on HN