Facebook Down?

Post by lovedswain » Thu, Apr 08, 2021, 10:36 PM UTC

erik_seaberg wrote: post #167

lovedswain wrote: post #102

Earlier quoted context omitted.

The global outage drinking game: bad DNS config push - 1 shot routing loop - 2 shots third party advertising your routes - 3 shots power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots Wolves ate through fiber - 5 shots And it was a full moon - 6 shots Single service failure, but service has not been restarted in 5 years, and no lo…

5 is often broken down to “rain fade” and “backhoe fade.” My favorite can’t-restart story is “the shared password manager is down, we need a hardware crypto key to restart it, the key is in the safe, and the safe’s combination was in the shared password manager.”

Post by austinheap » Thu, Apr 08, 2021, 10:40 PM UTC

londons_explore wrote: post #154

viraptor wrote: post #91

Earlier quoted context omitted.

At that size of a company there's usually blast radius restrictions and per-role permissions. I don't expect anyone has enough rights to "delete Facebook" on their own.

I guarantee you that there are 100+ people who could take Facebook down for 24+ hours if they went rogue. For example the people responsible for the bootup scripts of Facebook infra could sneak in a "0 0 1 * * /bin/rm -rf ${TEMPDIR}/*" into crontab... They'd set the commit message as "clear out temp monthly" and it would get deployed across the entire fleet till in the first of next month every disk at Facebook gets…

Post by londons_explore » Thu, Apr 08, 2021, 10:41 PM UTC

tracyhenry wrote: post #168

londons_explore wrote: post #154

Earlier quoted context omitted.

I guarantee you that there are 100+ people who could take Facebook down for 24+ hours if they went rogue. For example the people responsible for the bootup scripts of Facebook infra could sneak in a "0 0 1 * * /bin/rm -rf ${TEMPDIR}/*" into crontab... They'd set the commit message as "clear out temp monthly" and it would get deployed across the entire fleet till in the first of next month every disk at Facebook gets…

How though? Every past author of that script would be notified of such a change. It'd be insane if all of them would pretend they didn't see it and accept that change.

Post by ben509 » Thu, Apr 08, 2021, 10:42 PM UTC

mullingitover wrote: post #169

We're obviously going to take this opportunity to dunk of facebook because it's funny, but let's not forget that there are some valuable cultural assets that facebook hosts, with groups such as "Foods with threatening auras" and "Animals in predicaments." I long ago tamed the facebook engagement algorithm by simply unfollowing everything and everyone except these legitimate, useful types of entertainment venues.

Post by reaperducer » Thu, Apr 08, 2021, 10:43 PM UTC

erik_seaberg wrote: post #167

lovedswain wrote: post #102

Earlier quoted context omitted.

The global outage drinking game: bad DNS config push - 1 shot routing loop - 2 shots third party advertising your routes - 3 shots power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots Wolves ate through fiber - 5 shots And it was a full moon - 6 shots Single service failure, but service has not been restarted in 5 years, and no lo…

5 is often broken down to “rain fade” and “backhoe fade.” My favorite can’t-restart story is “the shared password manager is down, we need a hardware crypto key to restart it, the key is in the safe, and the safe’s combination was in the shared password manager.”