Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook, Instagram go down around the world in an apparent outage

usatoday.com

Post reply on HN
Print view View original

311–320 of 351 posts

Re: Facebook, Instagram go down around the world in an apparent outage

#311

Post by goblin89 »

Coincidentally, just watched The Social Network, the plot of which includes that quote by Mark:

> Let me tell you difference between Facebook and everybody else. We don't crash ever! If the serves are down for even a day, our entire reputation is irreversibly destroyed.

> Even a few people leaving would reverberate through the entire use base. The users are interconnected. That is the whole point. College kids are online because their friends are online, and if one domino goes, the other dominos go.

Re: Facebook, Instagram go down around the world in an apparent outage

#312

Post by ninth_ant »

goblin89 wrote: post #311

Coincidentally, just watched The Social Network, the plot of which includes that quote by Mark: > Let me tell you difference between Facebook and everybody else. We don't crash ever! If the serves are down for even a day, our entire reputation is irreversibly destroyed. > Even a few people leaving would reverberate through the entire use base. The users are interconnected. That is the whole point. College kids are on…

You’re quoting a movie, which played fast and loose with facts to tell a story (as movies do).

In real life, Facebook had significant issues with uptime in the early years.

Re: Facebook, Instagram go down around the world in an apparent outage

#313

Post by bpye »

hideo wrote: post #282
str33t_punk wrote: post #195

Earlier quoted context omitted.

Both companies are massive and have tons of developers. It becomes almost impossible to look at the system as a whole with the amount of changes coming through. And, you get scenarios where small failures cascade through the stack reaking havok. Often times its just one config change Its telling that one of the hottest areas of distributed systems research these days is the boring topic of configuration management. G…

>>Google, Microsoft, etc are paying researchers top dollar to figure out how to prevent massive outages through novel techniques Curious what makes you think this. Are there specific job postings in either company that are focused on this?

I work for Microsoft, I know of at least CrystalNet [1].

[1] - https://www.microsoft.com/en-us/research/blog/eliminating-ne...

Re: Facebook, Instagram go down around the world in an apparent outage

#314

Post by londons_explore »

chipperyman573 wrote: post #295
erobbins wrote: post #247

Earlier quoted context omitted.

No, FB datacenters are geographically diverse. They do run quarterly 'storms' where a datacenter is shut down to test failover and resiliency. I have no idea if today is one of those days, since I left last year.

Interesting. Out of curiosity, how hard is it to turn the datacenter "back on" in case they discover there's a problem with the failover?

Typically that could be done with one undrain command and take about 30 seconds.

Re: Facebook, Instagram go down around the world in an apparent outage

#316

Post by goblin89 »

ninth_ant wrote: post #312
goblin89 wrote: post #311

Coincidentally, just watched The Social Network, the plot of which includes that quote by Mark: > Let me tell you difference between Facebook and everybody else. We don't crash ever! If the serves are down for even a day, our entire reputation is irreversibly destroyed. > Even a few people leaving would reverberate through the entire use base. The users are interconnected. That is the whole point. College kids are on…

You’re quoting a movie, which played fast and loose with facts to tell a story (as movies do). In real life, Facebook had significant issues with uptime in the early years.

Can’t argue with that, however the fact that Facebook used to go down before would not preclude that each time was (and maybe still is) seen as a major incident within the original corporate culture.

Re: Facebook, Instagram go down around the world in an apparent outage

#317

Post by jomkr »

WrtCdEvrydy wrote: post #223
cheeze wrote: post #46

Earlier quoted context omitted.

Nothing worse than that sinking feeling of "oh fuck, we have to backfill a lot of data.

Why did my username on this site just change to 'test123'... oh, where clauses.

Nothing worse then the page on Friday night, oh there goes my weekend.

Re: Facebook, Instagram go down around the world in an apparent outage

#318

Post by AceyMan »

londons_explore wrote: post #314
chipperyman573 wrote: post #295

Earlier quoted context omitted.

Interesting. Out of curiosity, how hard is it to turn the datacenter "back on" in case they discover there's a problem with the failover?

Typically that could be done with one undrain command and take about 30 seconds.

Holy Batcave,

I'm a baby SRE compared to that .

Mind. Blown¹.

/Acey

¹—I don't use FB [true] but I hear it's huge :-P

Re: Facebook, Instagram go down around the world in an apparent outage

#319

Post by mclightning »

ceejayoz wrote: post #94
cronix wrote: post #90

It looks like something much larger is going on. If you look at the front page of https://downdetector.com/ you'll see most major sites/backbones are having issues (Verizon/ATT/Sprint/CenturyLink/TMobile/Comcast/Level3/etc).

That site relies on user reports. I strongly suspect users are reporting "my Internet is having troubles" because their FB, Messenger, etc. isn't working right. For example, in the comments of the T-Mobile outage page, there's stuff like "Haven't been able to upload anything to social media all day" and "Cannot send pictures through whatsapp and fb messenger".

Yes, I had that exact problem yesterday. I couldnt upload pictures/videos/gifs on whatsapp and instagram.

I thought maybe my ISP blocked a port which these services maybe transferring their multimedia on.

/Sweden

Re: Facebook, Instagram go down around the world in an apparent outage

#320

Post by z3t4 »

shafte wrote: post #177
snazz wrote: post #174

I’ve seen many systems go down over the last few days worldwide. Aside from the possibility of a mega-DDoS attack (which Facebook denies), all of these organizations have fairly diverse tech stacks to my knowledge. Google’s issue (supposedly) had to do with their Blobstore API, we don’t know what happened with Facebook, and many other, smaller services have had issues as well, including three intranet services at my…

The best explanation is coincidence, I think. I have direct knowledge of two of the incidents in the past few weeks, and they have completely unrelated causes. Sometimes you just get unlucky!

If it can happen, it will happen.
Post reply on HN