Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding how Facebook disappeared from the internet

blog.cloudflare.com

Post reply on HN
Print view View original

91–100 of 280 posts

Re: Understanding how Facebook disappeared from the internet

#91

Post by wyldfire »

OJFord wrote: post #57
huuuz wrote: post #55

Earlier quoted context omitted.

Reduced staff because of the pandemic? 18 months in? This looks like a sweet deal for those workers more than anything else.

Presumably they're working remotely, but the incident required physical access, so only the reduced number of people present were (immediately (or at all, I don't know their policies)) available to deal with it.

I read that on-site badge readers denied access to employees! (Not necessarily the critical team to resolving the outage, but still...)

Re: Understanding how Facebook disappeared from the internet

#92

Post by kelp »

foobarian wrote: post #74
Hikikomori wrote: post #48

Earlier quoted context omitted.

Restoring is just as simple as flipping the switch again, but access to that switch is another matter when your internal network is also down and you cannot even get access to your office or datacenters.

What I don't get is why there is no dead-man switch mechanism in place to roll back the configuration automatically unless someone confirms it positively. Kind of how screen resolution rolls back if you don't ack it. I used to always run a "(sleep 600; iptables -F) &" when messing with remote personal stuff just in case I lock myself out. I suppose with something like BGP it would be very difficult to get such a fall…

This is a key feature of Junos on Juniper devices. It's called 'commit confirmed' and it will apply the config and then auto-rollback if you don't confirm it within a certain amount of time.

https://www.juniper.net/documentation/us/en/software/junos/c...

But Facebook's network is certainly much more complex and automated than just doing one commit on one device.

But I think they do still uses Juniper devices at the edge.

Re: Understanding how Facebook disappeared from the internet

#93

Post by bsedlm »

nimbius wrote: post #64
throwdecro wrote: post #6

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

Unpopular opinion but I think a talent exodus and turnover are likely at fault here as well. The people who stood up this juggernaut are no longer here, and Facebooks ability to consistently attract talent that is competent enough to maintain such a formidable beast is hampered by repetitive revelations that it operates at the net-loss of humanity as a whole. Facebooks no longer an innovator, just a mining operation…

I'd say the same thing about Google.

Re: Understanding how Facebook disappeared from the internet

#94

Post by unityByFreedom »

zanethomas wrote: post #78

All very amusing. But most amusing was that facebook employees' badges wouldn't work. Hilarious. Can't get in the office if facebook.com is not reachable. (or whatever the unreachable service was)

Ah, so no playing oculus either?

Re: Understanding how Facebook disappeared from the internet

#95

Post by unityByFreedom »

Ansil849 wrote: post #72

Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it. Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing gl…

Not at all. It's interesting analysis. It's possible to bloviate in such a post but I don't see it here.

Re: Understanding how Facebook disappeared from the internet

#96

Post by threeseed »

Blackthorn wrote: post #83
Ansil849 wrote: post #72

Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it. Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing gl…

No surprises here. They do it all the time for websites they cdn. Cloudflare might as well be a marketing company for how much they puff about themselves.

Cloudflare is also incredibly dodgy with their billing practices which you wouldn't pick up from their marketing.

If you downgrade from a Paid to a Free account they automatically and silently add a paid add-on for 10 Page Rules.

Re: Understanding how Facebook disappeared from the internet

#99

Post by ninkendo »

pier25 wrote: post #98

This is all very interesting, but why couldn't FB just push an update remotely to solve this? I've seen mentions everywhere that to fix this they required physical access to the BGP routers.

How would they push an update if they couldn’t reach their own network?

My understanding is that if a bad BGP route was pushed and your whole network (typically accessed via VPN while the engineers WFH) is probably unreachable from any of you’re employees’ homes, it’s hard to get the right people on site to make a fix, or talk the remote hands folks at the date center through it over the phone. Troubleshooting is hard, troubleshooting while you have no access to your own network has got to be just that much harder.

Re: Understanding how Facebook disappeared from the internet

#100

Post by hinkley »

Ansil849 wrote: post #72

Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it. Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing gl…

They've done this in previous outages and people didn't like it then, either.

Generally big companies don't talk smack about other big companies having internet-wide issues, unless those issues are directly caused by the other company.

For instance, when Google talked about Cloudbleed, which was when Cloudflare vomited millions of secrets all over Google's caching heirarchy and Google had to manually clean it up.

I think perhaps the Cloudflare people have gotten confused and think that means it's okay to talk about other people's stuff. Instead of interpreting it as it really is, which is that Cloudflare is the last company that should be criticizing everyone else, lest someone bring up their previous missteps.

Post reply on HN