731–740 of 1001 posts
Earlier quoted context omitted.
My suspicion is that since a lot of internal comms runs through the FB domain and since everyone is still WFH, then its probably a massive issue just to get people talking to each other to solve the problem.
You mean the same problem as when GMail goes down and Googlers can't reach each other? I guess good decentralized public communication services could solve those issues for everybody.
Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…
Archived version: https://archive.is/QvdmH
Earlier quoted context omitted.
And there his account went poof, thanks for archiving.
They were quoted on multiple news sites including Ars Technica. I would imagine they were not authorized to post that information. I hope they don't lose their job. Shareholders and other business leaders I'm sure are much happier reporting this as a series of unfortunate technical failures (which I'm sure is part of it) rather than a company-wide organizational failure. The fact they can't physically badge in the pe…
Operations teams normally have a special room with a secure connection for situations like this, so that production can be controlled in the event of bgp failure, nuclear war, etc. I could see physical presence being an issue if their bgp router depends on something like a crypto module in a locked cage, in which case there's always helicopters.
So if anything, Facebook's labor policies are about to become cooler.
Ironic enough; status.fb.com also down.
Earlier quoted context omitted.
Seriously? Is that how it works?
No, the backbone of the internet is not maintained with patches sent in emails.
Earlier quoted context omitted.
Seriously? Is that how it works?
No, the backbone of the internet is not maintained with patches sent in emails.
In the post-mortem, we'll find out that Facebook's alerting and comms systems all run on Facebook. As a result, they can't even coordinate the restart to roll back changes.
Earlier quoted context omitted.
Seriously? Is that how it works?
No, the backbone of the internet is not maintained with patches sent in emails.