hahaha, good riddance
Facebook-owned sites were down
771–780 of 1001 posts
Re: Facebook-owned sites were down
#772There's still no connectivity to Facebook's DNS servers: > traceroute a.ns.facebook.com traceroute to a.ns.facebook.com (129.134.30.12), 30 hops max, 60 byte packets 1 dsldevice.attlocal.net (192.168.1.254) 0.484 ms 0.474 ms 0.422 ms 2 107-131-124-1.lightspeed.sntcca.sbcglobal.net (107.131.124.1) 1.592 ms 1.657 ms 1.607 ms 3 71.148.149.196 (71.148.149.196) 1.676 ms 1.697 ms 1.705 ms 4 12.242.105.110 (12.242.105.110)…
Looks like they misconfigured a web interface that they can't reach anymore now that they're off the net.
"anyone have a Cisco console cable lying around?"
Re: Facebook-owned sites were down
#773This event should be a good conversation starter on how horrifyingly monopolistic this trifecta of services has on worldwide communication. When I think through a random smattering of people in my contact book, I now have no way of contacting quite a few people at all. That's fucked. I wonder how many important messages, replies, etc will be screwed up due to this.
Re: Facebook-owned sites were down
#774Its always a DNS problem
Re: Facebook-owned sites were down
#775Okay, let me tell you the difference between Facebook and everyone else, we don't crash EVER! If those servers are down for even a day, our entire reputation is irreversibly destroyed! Users are fickle, Friendster has proved that. Even a few people leaving would reverberate through the entire userbase. The users are interconnected, that is the whole point. College kids are online because their friends are online, and…
At this point in time, I doubt this holds true.
Facebook is just too big and pervasive that such an outage would be treated by its users like an internet outage or a power outage. Once it's back online, everyone will forget.
Re: Facebook-owned sites were down
#776Earlier quoted context omitted.
My suspicion is that since a lot of internal comms runs through the FB domain and since everyone is still WFH, then its probably a massive issue just to get people talking to each other to solve the problem.
I don’t know how true it is but a few reports claim employees can’t get into the building with their badges.
[deleted]
Re: Facebook-owned sites were down
#777I would have thought that these companies that are richer then $GOD would have (virtual) instances of at least the previous stable version available for situations such as this. It would at least keep their damn doors open and internal communications systems going... Maybe they'll NOW think of such things? What's the cliche, penny wise and pound foolish? Or is it, no need to listen to experienced Network Designers? I…
most of what they do, they do with in house tools, and custom-everything, including hardware. as a consequence, for some classes of problems there are no experts - not at facebook, not anywhere.
i feel for their netops people. uncharted territory with the whole world watching and, no doubt, a lot of morons from management trying to be "helpful" in getting this nice crisis resolved. for any crisis there is always a bunch of clowns with MBAs that consider it their golden opportunity to shine (nearly always at someone elses expense)
Re: Facebook-owned sites were down
#778They made they own BGP tools and looks like it failed
https://www.youtube.com/watch?v=wHfYUbKNEyc
Re: Facebook-owned sites were down
#779Earlier quoted context omitted.
I don’t know how true it is but a few reports claim employees can’t get into the building with their badges.
I remember my first time having a meeting at Facebook and observing none of the doors had keyholes and thinking "hope their badge system never goes down"
Aaaaaaand it's down!
Re: Facebook-owned sites were down
#780Earlier quoted context omitted.
And there his account went poof, thanks for archiving.
They were quoted on multiple news sites including Ars Technica. I would imagine they were not authorized to post that information. I hope they don't lose their job. Shareholders and other business leaders I'm sure are much happier reporting this as a series of unfortunate technical failures (which I'm sure is part of it) rather than a company-wide organizational failure. The fact they can't physically badge in the pe…
> Edit: I also hope this doesn't damage prospects for more Work From Home. If they couldn't get anyone who knew the configuration in because they all live a plane ride away from the datacenters, I could see managers being reluctant to have a completely remote team for situations where clearly physical access was needed.
You're conflating working remotely ("a plane ride away") and working from home.
You're also conflating the people who are responsible network configuration, and for coming up with a plan to fix this; and the people who are responsible for physically interacting with systems. Regardless of WFH those two sets likely have no overlap at a company the size of Facebook.