61–70 of 351 posts
Google then Facebook and Instagram? My hunch is that it's the end of Q1 and people are trying to release code changes so they can pad their Q1 performance reviews "designed and delivered feature X on time in Q1".
"and brought down a service used by billions and losing potentially millions in revenue."
Serious question: Was any value lost? (this may appear sarcastic) Facebook obviously loses some ad revenue and Facebook customers may lose sales. But do Facebook/Instagram users suffer? But how does losing social media for several hours affect the quality of life of users?
Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate…
Look at how many service providers have increased incidents reported here: https://downdetector.com/ My bet is that people are having problems with FB/Insta and immediately assuming that the whole internet is messed up.
Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate…
Deny, deny, deny, obfuscate, deny, then blame someone else (usually, YOU).
What manner of failure would cause such globally deployed and distributed systems to go down like this? I'm very interested to read up on this when they release details of the failure.
Short duration: network, bad software deploy Long duration: db. If you break data, it takes a while to unbreak. Source: Me. My career has been spent managing db's for internet scale sites.
Screw ups related to data loss are rare (I've been here years and haven't seen one with the DBs that the stuff I work with uses) but failures at this scale tend to cascade a little ways and it takes time to dig out of the hole. They probably have the problem solved but they have to spend a bunch of time synchronizing things and verifying the fix before they press the big red "go live" button.