381–390 of 1001 posts
It's crazy that half the country doesn't have internet because Facebook stopped working.
Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…
Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue: > As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now…
> the people with physical access is separate from the people with knowledge of [...] Welcome to the brave new world of troubleshooting. This will seriously bite us one day.
Is anyone else seeing knock-on effects at the other major public DNS providers? I'm seeing nslookups sent to 4.2.2.2 and 8.8.8.8 intermittently timeout if the hostname does not belong to a major website. CloudFlare DNS (1.1.1.1) doesn't appear to be impacted. For example: [root@app ~]# nslookup downforeveryoneorjustme.com 4.2.2.2 ;; connection timed out; trying next origin ;; connection timed out; no servers could be…
"We're noticing an elevated level of usage for the time of day and are currently monitoring the performance of our systems. We do not anticipate this resulting in any impact to the service.
We have temporarily disabled typing notifications. We expect these to be re-enabled soon."
Earlier quoted context omitted.
I like how FB decided to send "ramenporn" as their spokesperson.
I'm worried about that person. I doubt Facebook will look kindly on breaking incident news being shared on reddit.
Earlier quoted context omitted.
It shouldn't be too stressful. Well-managed companies blame processes rather than people, and have systems set up to communicate rapidly when large-scale events occur. It can be sort of exciting, but it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. These resolutions are collaborative, shared efforts.
"it's not like there is one person typing at a keyboard with a hundred managers breathing down their neck. These resolutions are collaborative, shared efforts" Well, you'd be surprised about how one person can bring everything down and/or save the day at Facebook, Cloudflare, Google, Gitlab, etc. Most people are observers/cheerleaders when there is an incident.
Yeah, a typical fight/flight response.