161–170 of 208 posts
How on earth does EVERYTHING go down like this? It just seems unfathomable to me. Where is the single point of failure?
How does it stay up as much as it does :)
Likely not related, but you can imagine that they are quite busy these days with two unplanned projects: 1. Retooling the underpinnings of their ad analytics to survive iOS14. Big advertisers are about to lose a lot of their insights into attribution, partially directly related to the new limitations of iOS14, and partially due to assumptions that Facebook had made about tracking and which are no longer accurate and…
Earlier quoted context omitted.
If an intern can bring down the whole presence of your company online it's not the intern who is dumb.
While that's perfectly true, I'm sure a smart and determined intern could cause an astonishing amount of damage in many many orgs. Hopefully not at Facebook though.
Ditto if they can bring it down intentionally and cover their tracks so no one finds out who did it.
Earlier quoted context omitted.
The global outage drinking game: bad DNS config push - 1 shot routing loop - 2 shots third party advertising your routes - 3 shots power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots Wolves ate through fiber - 5 shots And it was a full moon - 6 shots Single service failure, but service has not been restarted in 5 years, and no lo…
You forgot: Hunter shot your fiber relay with a hunting rifle - 6 shots
How on earth does EVERYTHING go down like this? It just seems unfathomable to me. Where is the single point of failure?
The global outage drinking game: bad DNS config push - 1 shot routing loop - 2 shots third party advertising your routes - 3 shots power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots Wolves ate through fiber - 5 shots And it was a full moon - 6 shots Single service failure, but service has not been restarted in 5 years, and no lo…
My favorite can’t-restart story is “the shared password manager is down, we need a hardware crypto key to restart it, the key is in the safe, and the safe’s combination was in the shared password manager.”
Earlier quoted context omitted.
At that size of a company there's usually blast radius restrictions and per-role permissions. I don't expect anyone has enough rights to "delete Facebook" on their own.
I guarantee you that there are 100+ people who could take Facebook down for 24+ hours if they went rogue. For example the people responsible for the bootup scripts of Facebook infra could sneak in a "0 0 1 * * /bin/rm -rf ${TEMPDIR}/*" into crontab... They'd set the commit message as "clear out temp monthly" and it would get deployed across the entire fleet till in the first of next month every disk at Facebook gets…
I long ago tamed the facebook engagement algorithm by simply unfollowing everything and everyone except these legitimate, useful types of entertainment venues.
How on earth does EVERYTHING go down like this? It just seems unfathomable to me. Where is the single point of failure?
The global outage drinking game: bad DNS config push - 1 shot routing loop - 2 shots third party advertising your routes - 3 shots power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots Wolves ate through fiber - 5 shots And it was a full moon - 6 shots Single service failure, but service has not been restarted in 5 years, and no lo…