Did they move too fast and break too many things?
Facebook, Instagram go down around the world in an apparent outage
191–200 of 351 posts
Re: Facebook, Instagram go down around the world in an apparent outage
#192Earlier quoted context omitted.
Are you saying that a cold-war-era system like the internet/arpanet meant to survive a nuclear war might be vulnerable to an attack if we take all the code and data and store it in the same place? :-)
Cloud is not much different then Mainframe Computing 2.0. I'm already curious and have been pondering about this for quite a while to understand what/how/when we will see a disruption of this.
Limewire is gonna make a come back
Re: Facebook, Instagram go down around the world in an apparent outage
#193Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate…
So, a storm in Denver stops me from using Messengner in Estonia? I wonder where the butterfly flapped its wings.
Re: Facebook, Instagram go down around the world in an apparent outage
#194Whatever happens right now at Facebook is less important than the fact they will never say what affected them. Of course nobody would tell 'hey, outage right now due to 0day / mistake' but...
Re: Facebook, Instagram go down around the world in an apparent outage
#195I’ve seen many systems go down over the last few days worldwide. Aside from the possibility of a mega-DDoS attack (which Facebook denies), all of these organizations have fairly diverse tech stacks to my knowledge. Google’s issue (supposedly) had to do with their Blobstore API, we don’t know what happened with Facebook, and many other, smaller services have had issues as well, including three intranet services at my…
Both companies are massive and have tons of developers. It becomes almost impossible to look at the system as a whole with the amount of changes coming through. And, you get scenarios where small failures cascade through the stack reaking havok. Often times its just one config change
Its telling that one of the hottest areas of distributed systems research these days is the boring topic of configuration management. Google, Microsoft, etc are paying researchers top dollar to figure out how to prevent massive outages through novel techniques. It is one of the harder problems to solve and requires massive investment in tooling, refactoring, etc.
Re: Facebook, Instagram go down around the world in an apparent outage
#196Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate…
So, a storm in Denver stops me from using Messengner in Estonia? I wonder where the butterfly flapped its wings.
Pretty sure it doesn't apply to Facebook, but Amazon's cheapest AWS tiers are around there. Same with Virginia.
Re: Facebook, Instagram go down around the world in an apparent outage
#197Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate…
> "We are trying to gather information, we are making a bunch of client phone calls, we will know after we make those calls." I think that is a yes, and he getting ahead by saying "Yes and we have no idea why or ETA so let us do our job". Granted, they should have a status page.
Last time I dealt with a cloud provider outage the status page was unresponsive during the outage because the status page had some kind of dependency on the resources that were down...
Re: Facebook, Instagram go down around the world in an apparent outage
#198Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate…
Do they do this to get around and 99.9% uptime agreements?
It's easy to be cynical but it's optimistic expectation management.
It might be resolved, it has to get worse before you escalate it further. They might not know the full facts. Might be worse than it really is. How do you know? You can't judge that because your personal rendering of Facebook failed. You have load balancers and CDNs and A/B testers all getting in the way of delivering data to your machine.
It's too easy to draw a conclusion from the client-side armchair and the provider is absolutely not going to make false promises, for the worse or for the better.
You want to hope that Facebook, in this case, acts on more complete information.
Re: Facebook, Instagram go down around the world in an apparent outage
#199What manner of failure would cause such globally deployed and distributed systems to go down like this? I'm very interested to read up on this when they release details of the failure.
As someone who works at a large company in the networking space, you would be surprised that minor changes to configuration can cause catastrophic failures that are really challenging to come back from
Network failures are usually really bad when your system is globally deployed and distributed -- often times you can't even communicate with your machines to deliver fixes :p
Re: Facebook, Instagram go down around the world in an apparent outage
#200This is bigger than Facebook.
https://www.akamai.com/us/en/resources/visualizing-akamai/re...