Facebook, Instagram go down around the world in an apparent outage • Hacker News Board

Re: Facebook, Instagram go down around the world in an apparent outage

#291

Post by SilverSurfer972 » Thu, Mar 14, 2019, 3:24 AM UTC

edoceo wrote: post #215

SilverSurfer972 wrote: post #166

Earlier quoted context omitted.

That's the trust issue with current agreements we are solving. If an API is down the bound agreement is enforced instantly with our platform, no lies, no call, no pain. We are actually onboarding companies to try it out! https://stacktical.com

Even with your platform there will be calls and pain, maybe even lies during an outage. BC-Recordkeeping or not

unscheduled outages are always painful and people will always call, I agree. But instant compensation is doing a better job at damage control that a status page. Keeping customer satisfaction even in bad situation is key in a world of high availability expectations. And with a distributed, non partisan metric sourcing about the availability of an API, it's not possible for a Service Provider to lie anymore.

Feel free to give that whitepaper a look

Re: Facebook, Instagram go down around the world in an apparent outage

#292

Post by dontbenebby » Thu, Mar 14, 2019, 3:31 AM UTC

samstave wrote: post #258

popz41 wrote: post #58

Earlier quoted context omitted.

I imagine the NSA uses an optical tap device. These devices create identical copies and require no power or management.

And they have been using the USS Jimmy Carter sub with the front huge cable splice bay for decades to compromise all undersea cables. >>> The New York Times reported in 2005 that the USS Jimmy Carter, a highly advanced submarine that was the only one of its class built, had a capability to tap undersea cables. An Institute of Electrical and Electronics Engineers report speculated that a 45-foot extension added to the…

I wonder how Jimmy Carter feels about his namesake being used to wiretap the world?

https://www.washingtonpost.com/news/post-politics/wp/2014/03...

Re: Facebook, Instagram go down around the world in an apparent outage

#293

Post by zwayhowder » Thu, Mar 14, 2019, 3:33 AM UTC

hiccuphippo wrote: post #225

hiei wrote: post #52

Earlier quoted context omitted.

Gmail and other Google products went down last night. Close though. Thankfully not on Twitter or FB.

My company's tech support sent us an email to tell us our email was down. Fun times we live in.

Back in my tech support days I received an email from a customer "I am unable to send or receive emails" I replied "I am very sorry for your inconvenience, I have resolved the issue".

Customers in 1999 really couldn't believe no one had replied to their emails within a day or two.

Re: Facebook, Instagram go down around the world in an apparent outage

#294

Post by samstave » Thu, Mar 14, 2019, 3:42 AM UTC

dontbenebby wrote: post #292

samstave wrote: post #258

Earlier quoted context omitted.

And they have been using the USS Jimmy Carter sub with the front huge cable splice bay for decades to compromise all undersea cables. >>> The New York Times reported in 2005 that the USS Jimmy Carter, a highly advanced submarine that was the only one of its class built, had a capability to tap undersea cables. An Institute of Electrical and Electronics Engineers report speculated that a 45-foot extension added to the…

I wonder how Jimmy Carter feels about his namesake being used to wiretap the world? https://www.washingtonpost.com/news/post-politics/wp/2014/03...

It was a known FU to carter...

Re: Facebook, Instagram go down around the world in an apparent outage

#295

Post by chipperyman573 » Thu, Mar 14, 2019, 3:51 AM UTC

erobbins wrote: post #247

linsomniac wrote: post #53

Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate…

No, FB datacenters are geographically diverse. They do run quarterly 'storms' where a datacenter is shut down to test failover and resiliency. I have no idea if today is one of those days, since I left last year.

Interesting. Out of curiosity, how hard is it to turn the datacenter "back on" in case they discover there's a problem with the failover?

Re: Facebook, Instagram go down around the world in an apparent outage

#296

Post by apolloWBrazil » Thu, Mar 14, 2019, 3:59 AM UTC

T_D_K wrote: post #218

subcosmos wrote: post #162

Earlier quoted context omitted.

and autocorrect - it's doubleplus bad

*ungood

They could be consolidating all of the DB infrastructure for their platforms. A zero down time dial-up would not be possible as they would need to nearly double their DB infrastructure. Short planned temporary outages of various features probably become long unplanned cross-platform outages. They probably decided to not rollback the migration after the first outage.

Re: Facebook, Instagram go down around the world in an apparent outage

#297

Post by macintux » Thu, Mar 14, 2019, 4:13 AM UTC

duado wrote: post #261

mtw wrote: post #190

Earlier quoted context omitted.

Good question is why oh why switch WhatsApp to Facebook tech when it was running perfectly ok on its own. Never crashed.

So that engineers can be moved between product groups while carrying relevant knowledge and experience with them.

If that’s the goal, why not start incorporating more Erlang into the rest of Facebook instead? It proved its mettle at WhatsApp.

Re: Facebook, Instagram go down around the world in an apparent outage

#298

Post by calebegg » Thu, Mar 14, 2019, 4:13 AM UTC

ceejayoz wrote: post #114

cronix wrote: post #111

Earlier quoted context omitted.

Look at the scale. The blue area is 100% below normal. The red are way above normal.

That doesn't make any sense, given that the "traffic" tab's scale says "7% above normal". The red are the areas with the most attacks, and as you'd expect, they correspond to large population centers. (It's also not very granular, and appears to largely correspond to "where does Akamai have a datacenter".)

https://xkcd.com/1138/

Re: Facebook, Instagram go down around the world in an apparent outage

#299

Post by B-Con » Thu, Mar 14, 2019, 4:13 AM UTC

40acres wrote: post #172

I'm interviewing for a Production Engineer role at Facebook on Monday, thanks for providing relevant "do you have any questions for us" content.

When I interviewed for SRE at Google, they'd had a non-trivial cross product outage days before. Good conversation starter, but I couldn't get many details out of them.

Re: Facebook, Instagram go down around the world in an apparent outage

#300

Post by duado » Thu, Mar 14, 2019, 4:22 AM UTC

macintux wrote: post #297

duado wrote: post #261

Earlier quoted context omitted.

So that engineers can be moved between product groups while carrying relevant knowledge and experience with them.

If that’s the goal, why not start incorporating more Erlang into the rest of Facebook instead? It proved its mettle at WhatsApp.

Facebook has to hire thousands of engineers per year. They may incorporate more Erlang into Facebook, but they have to have a core tech stack that can easily onboard engineers from a variety of backgrounds. I don't have the foggiest idea of whether Erlang can be part of that or not, but people talk about it as if it's a special-purpose tool.