Facebook employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors.
Facebook-owned sites were down
721–730 of 1001 posts
Re: Facebook-owned sites were down
#722Earlier quoted context omitted.
I like how FB decided to send "ramenporn" as their spokesperson.
A particular facet I love of the internet era is journalists reporting serious events while having to use the completely absurd usernames... "A Facebook engineer in the response team, ramenporn..."
My favorite example of this is when I saw references to "Goatse Security" on the front page of the Wall Street Journal
Re: Facebook-owned sites were down
#723Earlier quoted context omitted.
Interesting that they published stuff about their BGP setup and infrastructure a few months ago - maybe a little tweak to roll backs is needed. "... We demonstrate how this design provides us with flexible control over routing and keeps the network reliable. We also describe our in-house BGP software implementation, and its testing and deployment pipelines. These allow us to treat BGP like any other software componen…
Surely Facebook don't update routing systems between data centres (IIRC the situation) when they don't have people present to fix things if they go wrong? Or have an out-of-band connection (satellite, or dial-up (?), or some other alternate routing?). I must be misunderstanding this situation here. [Aside: I recall updating wi-fi settings on my laptop and first checking I had direct Ethernet connection working ... an…
Move fast and break
.
.
.
Re: Facebook-owned sites were down
#724Aren't there places around poorer countries where Facebook is basically an ISP? What about them?
https://tcrn.ch/3kOHco1 2Africa cable, as an example
Re: Facebook-owned sites were down
#725Earlier quoted context omitted.
> the people with physical access is separate from the people with knowledge of [...] Welcome to the brave new world of troubleshooting. This will seriously bite us one day.
folks with physical access are also denied. source - https://twitter.com/YourAnonOne/status/1445100431181598723
Let me guess, it is tied to FB systems which are down. That would be hilarious.
Re: Facebook-owned sites were down
#726Earlier quoted context omitted.
My suspicion is that since a lot of internal comms runs through the FB domain and since everyone is still WFH, then its probably a massive issue just to get people talking to each other to solve the problem.
You mean the same problem as when GMail goes down and Googlers can't reach each other? I guess good decentralized public communication services could solve those issues for everybody.
Googler here - my opinions are my own, not representing the company
at the lowest level in case of severe outage we resort to IRC, Plain Old Telephone Service and, sometimes, stick-it notes taped to windows...
Re: Facebook-owned sites were down
#727Earlier quoted context omitted.
My suspicion is that since a lot of internal comms runs through the FB domain and since everyone is still WFH, then its probably a massive issue just to get people talking to each other to solve the problem.
I don’t know how true it is but a few reports claim employees can’t get into the building with their badges.
Re: Facebook-owned sites were down
#728Perhaps tomorrow, the brave man or woman responsible for this beautiful screw up will step forward in HN for an outstanding ovation. Whoever did this, thank you! As a souvenir I took a screenshot on my phone.
that's not how post-mortems work
Re: Facebook-owned sites were down
#729Earlier quoted context omitted.
Seriously? Is that how it works?
I don't know. I doubt. It's just funny to think that you need email to fix BGP, but DNS is down because of BGP. You need DNS to send email which needs BGP. It's a kind of chicken and egg problem but at a massive scale this time.
Sheera Frenkel:
Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors.
Re: Facebook-owned sites were down
#730[deleted]