Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook was down

facebook.com

Post reply on HN
Print view View original

211–220 of 294 posts

Re: Facebook was down

#211

Post by jsmthrowaway »

nostrademons wrote: post #208
mjdesa wrote: post #193

Earlier quoted context omitted.

SREs?

Site Reliability Engineer. It's a Google (+Facebook)-specific title that is sort of like a sysadmin or devops, but instead of keeping the system up, they write code that keeps the system up. They also have a different negotiating position vs. engineering than in many other companies, eg. SREs have veto power over many architectural decisions in the code, and it's more "we'll build the system that can stay upright wit…

Apple hires SREs who are actually sysadmins that occasionally code, complicating the title somewhat. This is not unique to them.

nostrademons's explanation of SRE is the correct one, IMO. The architecture is key. Engineering has to be built to allow that. It has helped me in the past to say SREs are concerned more with the operation of a service than a group of machines offering a service; it's almost like a service operations developer. When a company thinks in terms of services and abstracts the machine away, i.e., containers, scheduling, Mesos, Omega/, intelligent CI/CD, service discovery, now you're getting into SRE territory instead of SA territory. The architecture involvement distinguishes SRE from devops for me. You should be able to trust SRE to build services, not just run engineering output.

Teams that congeal out of Xooglers tend to preach SRE well, and there is the occasional company (Twitter and Foursquare come to mind) that applies the title and interacts with the team as intended.

Re: Facebook was down

#212

Post by flexie »

blazespin wrote: post #201
eridius wrote: post #173

Earlier quoted context omitted.

I gotta say I'm finding it hard to accept their word that they're responsible for Facebook. I haven't paid much attention but I'm under the impression that they primarily just deal in DDoS's and other crude attacks, and I have a hard time imagining that they could cause a large enough DDoS to affect the massive juggernaut that is Facebook. Especially since Facebook just came back up and is now perfectly responsive an…

Simple Bayesian analysis says it is them as they do not have a record of lying. Why would they? There is nothing to gain and everything to lose.

How does simple Bayesian analysis suggest that? :-)

Re: Facebook was down

#214

Post by eridius »

thrownaway2424 wrote: post #200
eridius wrote: post #173

Earlier quoted context omitted.

I gotta say I'm finding it hard to accept their word that they're responsible for Facebook. I haven't paid much attention but I'm under the impression that they primarily just deal in DDoS's and other crude attacks, and I have a hard time imagining that they could cause a large enough DDoS to affect the massive juggernaut that is Facebook. Especially since Facebook just came back up and is now perfectly responsive an…

Yeah, it doesn't seem like a DoS since facebook was reachable and serving error pages, and instagram was reachable and serving blank pages. It _does_ seem like an intrusion or other security incident though because I'd be surprised to learn that instagram shares lots of critical infrastructure with facebook. It seems more likely that someone hit the panic button for both sites.

Some sort of security incidence does seem more plausible than a DDoS, although I can't think of what would affect Facebook, Instagram, Tinder, Hipchat, and AIM simultaneously. I'm also having a hard time imagining what sort of security incident would result in Facebook deliberately shutting down their web presence, even for a few minutes. And all of the other potential attacks, such as DNS or CDN, seems like it a) wouldn't affect everyone simultaneously, and b) wouldn't even work because sites like Facebook don't have a single point of failure, there's always backups and backups for the backups.

Re: Facebook was down

#215

Post by peteretep »

eridius wrote: post #173
bombtrack wrote: post #92

Obviously anyone can "take credit" for things like this, but if this[1] is true, then jesus. Edit: Since it's topical, I enjoy listening to Chuck Rossi in interviews or presentations. Releng 2014 - Keynote 1: Chuck Rossi, Release Engineering, Facebook Inc. | Talks at Google [2] [1] https://twitter.com/lizardmafia/status/559963134006292481 [2] http://youtu.be/Nffzkkdq7GM?t=4m39s

I gotta say I'm finding it hard to accept their word that they're responsible for Facebook. I haven't paid much attention but I'm under the impression that they primarily just deal in DDoS's and other crude attacks, and I have a hard time imagining that they could cause a large enough DDoS to affect the massive juggernaut that is Facebook. Especially since Facebook just came back up and is now perfectly responsive an…

    > they could cause a large enough DDoS to affect the
    > massive juggernaut that is Facebook
You don't need to send Facebook more traffic than it normally gets, you simply need to nail a pain point, and cause an unusual traffic event.

There's (I assume) no chance that it's been done using Slow Loris, but that's a good example of doing something unusual to deplete a resource in an unusual way.

I would tend to agree that it's unlikely Facebook have simply been flooded off the internet, but there are many other ways to perform much more targeted DDoS attacks, and presumably Facebook haven't mitigated against _all_ of them.

Re: Facebook was down

#216

Post by obituary_latte »

IvyMike wrote: post #93
obituary_latte wrote: post #68

Earlier quoted context omitted.

Restarting your modem is step 1 in troubleshooting internet connectivity problems when Facebook is down?

It's almost like he mentioned it because he realized it was silly after the fact and wanted to share that bit of mirth with us.

Or Facebook is just that important. We will never know for sure.

Re: Facebook was down

#217

Post by peteretep »

eridius wrote: post #207
blazespin wrote: post #201

Earlier quoted context omitted.

Simple Bayesian analysis says it is them as they do not have a record of lying. Why would they? There is nothing to gain and everything to lose.

Nothing to gain? Taking down Facebook seems like something that would bolster their rep significantly. If they think they can claim credit for it and get away with it, then there seems to be little reason why they wouldn't. I guess we'll just have to wait and see if Facebook makes any public statements as to what the cause of the outage was.

    > seems like something that would bolster their rep
    > significantly
And then irretrievably kill it if they were shown to be lying.

Re: Facebook was down

#218

Post by adhipg »

overworkedasian wrote: post #132

This doesnt look right: whois facebook.com @8.8.8.8 Whois Server Version 2.0 Domain names in the .com and .net domains can now be registered with many different competing registrars. Go to http://www.internic.net for detailed information. FACEBOOK.COM.DISABLE.YOUR.TIMELINE.NOW.WITH.THE.ORIGINAL.TIMELINE-REMOVE.NET FACEBOOK.COM.GET.ONE.MILLION.DOLLARS.AT.WWW.UNIMUNDI.COM FACEBOOK.COM.LOVED.BY.WWW.SHQIPHOST.COM FACEBOO…

You get the info that you're looking for if you do

whois =facebook.com

Re: Facebook was down

#219

Post by eridius »

peteretep wrote: post #215
eridius wrote: post #173

Earlier quoted context omitted.

I gotta say I'm finding it hard to accept their word that they're responsible for Facebook. I haven't paid much attention but I'm under the impression that they primarily just deal in DDoS's and other crude attacks, and I have a hard time imagining that they could cause a large enough DDoS to affect the massive juggernaut that is Facebook. Especially since Facebook just came back up and is now perfectly responsive an…

> they could cause a large enough DDoS to affect the > massive juggernaut that is Facebook You don't need to send Facebook more traffic than it normally gets, you simply need to nail a pain point, and cause an unusual traffic event. There's (I assume) no chance that it's been done using Slow Loris, but that's a good example of doing something unusual to deplete a resource in an unusual way. I would tend to agree that…

Good point. Although I'd say that any such attack like that is still unlikely to cause the entire site to become instantly unresponsive (or to serve static error pages as other people have suggested; personally, I just never got the site to load). And similarly, it would not be expected to come back to normal operating behavior a few minutes later. So yes, I'll grant you that you can DDoS even large sites without needing pure brute traffic, but it still doesn't seem particularly likely here.
Post reply on HN