691–700 of 1001 posts
Earlier quoted context omitted.
I like how FB decided to send "ramenporn" as their spokesperson.
A particular facet I love of the internet era is journalists reporting serious events while having to use the completely absurd usernames... "A Facebook engineer in the response team, ramenporn..."
My bbs handle from 30 years ago.
Not only facebook, but also Google, Zoom, Telegram, Youtube and many more internet service/ product/ providers from 8:00 AM today. This is more like internet outage.
Earlier quoted context omitted.
There are various non-FB fallback measures, including IRC as a last-ditch method. The IRC fallback is usually tested once a year for each engineer.
Good planning! Now, where does the IRC server live, and is it currently routable from the internet? While normally I know the advice is "Don't plan for mistakes not to happen, it's impossible, murphy's law, plan for efficient recovery for mistakes"... when it comes to "literally our entire infrastructure is no longer routable from the internet", I'm not sure there's a great alternative to "don't let that happen. ever…
Routing is one thing which you can't do without (then you need to fallback to phone communications), but DNS is something that's quite probable to not work well in a major disaster.
It is about time to speculate about sabotage, a disgruntled employee or something more exotic. All this BGP talk is boring.
Signal is welcoming everyone: https://nitter.mailstation.de/signalapp/status/1445062426739...
Earlier quoted context omitted.
And there his account went poof, thanks for archiving.
They were quoted on multiple news sites including Ars Technica. I would imagine they were not authorized to post that information. I hope they don't lose their job. Shareholders and other business leaders I'm sure are much happier reporting this as a series of unfortunate technical failures (which I'm sure is part of it) rather than a company-wide organizational failure. The fact they can't physically badge in the pe…
I hope they do.
#1 it's a clear breach of corporate confidentiality policies. I can say that without knowing anything about Facebook's employment contracts. Posting insider information about internal company technical difficulties is going to be against employment guidelines at any Big Co.
In a situation like this that might seem petty and cagey. But zooming out and looking at the bigger picture, it's first and foremost a SECURITY issue. Revealing internal technical and status updates needs to go through high-level management, security, and LEGAL approvals, lest you expose the company to increased security risk by revealing gaps that do not need to be publicized.
(Aside: This is where someone clever might say "Security by obscurity is not a strategy". It's not the ONLY strategy, but it absolutely is PART of an overall security strategy.)
#2 just purely from a prioritization/management perspective, if this was my employee, I would want them spending their time helping resolve the problem not post about it on reddit. This one is petty, but if you're close enough to the issue to help, then help. And if you're not, don't spread gossip - see #1.
Earlier quoted context omitted.
It shouldn't be too stressful. Well-managed companies blame processes rather than people, and have systems set up to communicate rapidly when large-scale events occur. It can be sort of exciting, but it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. These resolutions are collaborative, shared efforts.
> It can be sort of exciting, but it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. As someone who formerly did Ops for many many years... this is not accurate. Even in a well organized company there are usually stakeholders at every level on IM calls so that they don't need to play "telephone" for status. For an incident of this size, it wouldn't be unusual to…
Unsurprisingly, Oculus is down as well, as are most services for the VR headset. So that's 4 major properties right now.