Live data from Hacker News

Facebook-owned sites were down

facebook.com

991–1000 of 1001 posts

Re: Facebook-owned sites were down

#991

There's still no connectivity to Facebook's DNS servers: > traceroute a.ns.facebook.com traceroute to a.ns.facebook.com (129.134.30.12), 30 hops max, 60 byte packets 1 dsldevice.attlocal.net (192.168.1.254) 0.484 ms 0.474 ms 0.422 ms 2 107-131-124-1.lightspeed.sntcca.sbcglobal.net (107.131.124.1) 1.592 ms 1.657 ms 1.607 ms 3 71.148.149.196 (71.148.149.196) 1.676 ms 1.697 ms 1.705 ms 4 12.242.105.110 (12.242.105.110)…

Heck of a coincidence I must say... I can imagine this affects many other sites that use FB for authentication and tracking. If people pay proper attention to it, this is not just an average run of the mill "site outage", and instead of checking on or worrying about backups of my FB data (Thank goodness I can afford to lose it all), I'm making popcorn... Hopefully law makers all study up and pay close attention. What…

Indeed, what happened shows a good reason not to rely only on social log-in for various sites.

Re: Facebook-owned sites were down

#992
post #968

Earlier quoted context omitted.

Its partially there. C and D are still not in the global tables according to routeviews ie. 185.89.219.12 is still not being advertised to anyone. My peers to them in Toronto have routes from them, but not sure how far they are supposed to go inside their network. (past hop 2 is them) % traceroute -q1 -I a.ns.facebook.com traceroute to a.ns.facebook.com (129.134.30.12), 64 hops max, 48 byte packets 1 torix-core1-10G…

Oh my gosh, their IPv6 address contains "face:b00c"... > 2a03:2880:f0fc:c:face:b00c:0:35

How difficult is to get such a "vanity" address?

Re: Facebook-owned sites were down

#993

Earlier quoted context omitted.

My suspicion is that since a lot of internal comms runs through the FB domain and since everyone is still WFH, then its probably a massive issue just to get people talking to each other to solve the problem.

You mean the same problem as when GMail goes down and Googlers can't reach each other? I guess good decentralized public communication services could solve those issues for everybody.

BGP is decentralised.

Re: Facebook-owned sites were down

#994

There's still no connectivity to Facebook's DNS servers: > traceroute a.ns.facebook.com traceroute to a.ns.facebook.com (129.134.30.12), 30 hops max, 60 byte packets 1 dsldevice.attlocal.net (192.168.1.254) 0.484 ms 0.474 ms 0.422 ms 2 107-131-124-1.lightspeed.sntcca.sbcglobal.net (107.131.124.1) 1.592 ms 1.657 ms 1.607 ms 3 71.148.149.196 (71.148.149.196) 1.676 ms 1.697 ms 1.705 ms 4 12.242.105.110 (12.242.105.110)…

Can someone explain why it is also down when trying to access it via Tor using its onion address: http://facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5t... Or when trying ips directly: https://www.lifewire.com/what-is-the-ip-address-of-facebook-... I would have expected a DNS issue to not affect either of these. I can understand the onionsite being down if facebook implemented it the way a thirdparty would (a p…

routing was down _everywhere_ so tor is getting a better experience than most people by getting a 500 error

Re: Facebook-owned sites were down

#995

Earlier quoted context omitted.

The account has been deleted as well.

What are they afraid of? While they are sharing information that's internal/proprietary to the company, it isn't anything particularly sensitive and having some transparency into the problem is good for everyone. Who'd want to work for a company that might take disciplinary action because an SRE posted a reddit comment to basically say "BGP's down lol" - If I was in charge I'd give them a modest EOY bonus for being h…

They got told, explicitly that they shouldn't be sharing updates from the outage meeting, in the outage meeting.

Re: Facebook-owned sites were down

#996
post #968

Earlier quoted context omitted.

Oh my gosh, their IPv6 address contains "face:b00c"... > 2a03:2880:f0fc:c:face:b00c:0:35

How difficult is to get such a "vanity" address?

You just need to get a large enough block so that you can throw most of it away by adding your own vanity part to the prefix you are given. IPv6 really isn't scarce so you can actually do that.

Re: Facebook-owned sites were down

#997

Facebook down, WhatsApp down, but Signal still works. Time for a change? EDIT: Yes, Signal is not federated, but that's what people are at least ready to consider as a WhatsApp alternative. I also created Matrix / Element account, and had 0 contacts using it already.

The reason contacts don't tend to show up on Matrix/Element is because we don't push the user into sharing their addressbooks given the obvious privacy issues. Instead you mainly have to figure out who you know out-of-band for now (e.g. tweet "hey, who's on Matrix?").

I would be happy to have an option to share my availability on Matrix with other people that decided the same if that would mean I could bootstrap my network on that platform.

As it is now, Matrix may offer better privacy and more robust weil federated and p2p, but if I had to personally ask all of my contacts if they actually use it using some other medium, I can keep using that other medium too for conversations.

Re: Facebook-owned sites were down

#999
My father (https://bit.ly/3acZAAI), who is a certified CCSP Ethical Hacker and formerly worked @ZScaler/Checkpoint/Palo Alto Networks, would say that there are basically two scenarios: someone like him did it intentionally or someone like him did it by mistake.

Any other scenario of outsiders, code updates, etc - basically misses the point of how modern DNS infrastructure works.

Re: Facebook-owned sites were down

#1000

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…

> Reddit r/Sysadmin user that claims to be on the "Recovery Team" They have time to make public posts, and think it's a good idea? Sure, I'm on the 'Recovery Team' too! How about you?

If it's anything like my past employers, they probably have a lot of time. They probably also got in a lot of trouble.

When we'd have situation bridges put in place to work a critical issue, there would usually be 2-3 people who were actively troubleshooting and a bunch of others listening in, there because "they were told to join" but with little-to-nothing to do. In the worst cases, there was management there, also.

Most of the time I was one of the 2 or 3 and generally preferred if the rest of them weren't paying much attention to what was going on. It's very frustrating when you have a large group of people who know little about what's going on injecting their opinions while you're feverishly trying to (safely) resolve a problem.

It was so bad that I once announced[0] to a C-Level and VP that they needed to exit the bridge, immediately because the discussion devolved into finger-pointing. All of management was "kicked out". We were close to solving it but technical staff was second-guessing themselves in the presence of folks with the power to fire them. 30 minutes later we were working again. My boss at the time explained that management created their own bridge and the topic was "what do to about 'me'" which quickly went from "fire me" to "get them all a large Amazon gift card". Despite my undiplomatic handling of the situation, that same C-Level negotiated to get me directly beneath during a reorganization about six months later and I stayed in that spot for years with a very good working relationship. One of my early accomplishments was to limit any of management's participation in situation bridges to once/hour, and only when absolutely necessary, for status updates assuming they couldn't be gotten any other way (phones always worked, but the other communication options may not have).

[0] This was the 16th hour of a bridge that started at 11:00 PM after a full work day early in my career -- I was a systems person with a title equivalent to 'peon', we were all very raw by then and my "announcement" was, honestly, very rude, which I wasn't proud of. Assertive does not have to be rude, but figuring out the fine line between expressing urgency and telling people off is a skill that has to be learned.

Post reply on HN