Live data from Hacker News

Facebook, Instagram go down around the world in an apparent outage

usatoday.com

251–260 of 351 posts

Re: Facebook, Instagram go down around the world in an apparent outage

#251
post #111

Earlier quoted context omitted.

The attacks tab says the current count is "100% Below normal". It does not support what you think it supports. It "lights up" because there are always attacks happening.

Look at the scale. The blue area is 100% below normal. The red are way above normal.

That chart is a marketing tool, nothing more.

Re: Facebook, Instagram go down around the world in an apparent outage

#253
post #230

Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate…

"The storm"? It's sunny in the Bay Area for the first time in I don't know how long. I imagine it's nice in other parts of the world as well, other than where this localized "storm" is.

You missed the bomb cyclone that's happening right now in half the country?

Re: Facebook, Instagram go down around the world in an apparent outage

#254

Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate…

No, FB datacenters are geographically diverse. They do run quarterly 'storms' where a datacenter is shut down to test failover and resiliency. I have no idea if today is one of those days, since I left last year.

Theoretically a real shutdown might go in a different way than previous tests or simulations. For instance in a test you might cut the connection completely, while in the real case only some power circuits go down or whatever.

For instance GitHub's relatively recent shutdown was due to a fail-over heartbeat not going as expected.

Re: Facebook, Instagram go down around the world in an apparent outage

#256

What manner of failure would cause such globally deployed and distributed systems to go down like this? I'm very interested to read up on this when they release details of the failure.

Here's one example https://rachelbythebay.com/w/2019/01/20/quiet/

Re: Facebook, Instagram go down around the world in an apparent outage

#257

Earlier quoted context omitted.

Facebook is a paid advertising platform. There are plenty of people not getting what they paid for as a result of an outage.

ok but you are all talking about the users. facebook platform is free to use for developers as for the advertisers, i doubt they 'll be charged for impressions that didn't happen

But for someone who runs a business that relies on Instagram for marketing, and pays for advertising on that platform, it's a bit scary when the whole thing is down. Obviously this was only temporary this morning, and sure, no charges while down, but doesn't do me much good...

Re: Facebook, Instagram go down around the world in an apparent outage

#258
post #58

Earlier quoted context omitted.

Private post Morten: The NSA middleware we are required to run (that took time to deploy to each of our social partners) is breaking something so let’s revert. Public post Mortem: Entirely believable technical cause.

I imagine the NSA uses an optical tap device. These devices create identical copies and require no power or management.

And they have been using the USS Jimmy Carter sub with the front huge cable splice bay for decades to compromise all undersea cables.

>>>The New York Times reported in 2005 that the USS Jimmy Carter, a highly advanced submarine that was the only one of its class built, had a capability to tap undersea cables. An Institute of Electrical and Electronics Engineers report speculated that a 45-foot extension added to the Jimmy Carter provided this capability by allowing engineers to bring the cable up into a floodable chamber to install a tap. But it is unlikely that the USS Jimmy Carter routinely taps cables since U.S. intelligence agencies can much more easily (and lawfully) obtain cable data through taps at above-ground cable landing stations.

https://www.lawfareblog.com/evaluating-russian-threat-unders...

Re: Facebook, Instagram go down around the world in an apparent outage

#259

Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate…

A company as large and sophisticated as FB has data centers and cloud services in multiple countries, and in the US, probably colocated data centers. Certainly nothing localized to where you are.

If the outage is at all infrastructure-related, the root cause was something that at some point was local and cascaded. Unless someone git pushed to a repo used by both companies and it's taken all day to get it git revert'd, their redundancy obviously didn't work, did it? There's effectively a category-2 hurricane moving from the Rockies through the mid-west right now.

Re: Facebook, Instagram go down around the world in an apparent outage

#260
post #59

Serious question: Was any value lost? (this may appear sarcastic) Facebook obviously loses some ad revenue and Facebook customers may lose sales. But do Facebook/Instagram users suffer? But how does losing social media for several hours affect the quality of life of users?

FB has effectively replaced all other text messaging for several of my social circles. It's nice when you have groups that kinda change over time, otherwise group-texts always end up with numbers you don't necessarily have in your phone, etc.
Post reply on HN