Official update: "Facebook and Instagram experienced a major outage tonight from 22:10 until 23:10 PST. Our engineers identified the cause of the outage and recovered the site quickly. You should now see decreasing error rates while our systems stabilize. We don't expect any other break in service. I'll post another update within 30 mins. Thank you for your patience."
https://developers.facebook.com/status/issues/39399836411226...
Facebook was down
231–240 of 294 posts
Re: Facebook was down
#232Earlier quoted context omitted.
I gotta say I'm finding it hard to accept their word that they're responsible for Facebook. I haven't paid much attention but I'm under the impression that they primarily just deal in DDoS's and other crude attacks, and I have a hard time imagining that they could cause a large enough DDoS to affect the massive juggernaut that is Facebook. Especially since Facebook just came back up and is now perfectly responsive an…
Yeah, normal access patterns to FB are already pretty much as big as a DDOS on a regular website
For their service infrastructure, certainly. Napkin math suggest that they have to sustain peak floods in the ballpark of 500M concurrent clients. Give or take 100M.
However, if you could cripple DNS propagation/resolution from their systems to the next-hop upstreams, it would show up as services being very slow to respond. They do serve content from multiple domains.
I'd be interested to know how Google's 8.8.8.8 infrastructure defends them from DNS level manipulation...
Re: Facebook was down
#233Earlier quoted context omitted.
Would be fascinated to see them take on big G. E: not that I support these guys in any way. Just curious what would happen -- and whether we might all learn something about DDoS mitigation from such an event.
Google gets DDoSed all the time. You don't hear about it because, well, the DDoS SREs are very, very good at what they do.
In recent Google SREs AMA, they talked about how they mitigate DDoS attacks.
http://www.reddit.com/r/IAmA/comments/177267/we_are_the_goog...
http://www.reddit.com/r/IAmA/comments/177267/we_are_the_goog...
Re: Facebook was down
#234Re: Facebook was down
#235Re: Facebook was down
#236Earlier quoted context omitted.
I'm wondering how much revenue gets lost every minute that Facebook is down. It has to be in the tens of thousands. Edit: someone wrote this below in the comments, which came to $400 per second of downtime. Ouch.
For comparison, the cost of downtime for a single oil rig is on the order of $100 per second. Petrobras, as an example, operates 70 rigs and has a market cap of $125 billion. Facebook has a market cap of $216 billion. (source for cost: Cormorant Alpha shutdown in 2013) Edit: misplaced parenthesis => my first number was way off, $70 000 vs $100. Redid analysis, made mental note not to do math before coffee intake.
That seems a tad... high.
$70,000 = $2 trillion a year, assuming 24/365 usage.
Total value of all oil produced in the world (assuming $50/barrel) is roughly $1.4 trillion a year.
Re: Facebook was down
#237Earlier quoted context omitted.
I'm wondering how much revenue gets lost every minute that Facebook is down. It has to be in the tens of thousands. Edit: someone wrote this below in the comments, which came to $400 per second of downtime. Ouch.
For comparison, the cost of downtime for a single oil rig is on the order of $100 per second. Petrobras, as an example, operates 70 rigs and has a market cap of $125 billion. Facebook has a market cap of $216 billion. (source for cost: Cormorant Alpha shutdown in 2013) Edit: misplaced parenthesis => my first number was way off, $70 000 vs $100. Redid analysis, made mental note not to do math before coffee intake.
http://www.offshoreenergytoday.com/cormorant-alpha-shutdown-... estimates that the Cormorant Alpha shutdown cost $10,000,000/day, or about $100/second. That figure comes from multiplying the oil production (90,000 barrels/day) by the oil price ($110/barrel). Where did you get the $70,000/second figure from?
Re: Facebook was down
#238Re: Facebook was down
#239Official update: "Facebook and Instagram experienced a major outage tonight from 22:10 until 23:10 PST. Our engineers identified the cause of the outage and recovered the site quickly. You should now see decreasing error rates while our systems stabilize. We don't expect any other break in service. I'll post another update within 30 mins. Thank you for your patience." https://developers.facebook.com/status/issues/393…
"The issue was resolved at 23:10 PST and the site stabilized shortly afterwards. Our internal and external monitoring shows that API requests are being served with normal latency and error rates, in all geographic regions. We are sorry for any inconvenience this may have caused you and the users of your apps."
No hints there, guess we'll just have to wait for a full post-mortem to come out.
Re: Facebook was down
#240Earlier quoted context omitted.
For comparison, the cost of downtime for a single oil rig is on the order of $100 per second. Petrobras, as an example, operates 70 rigs and has a market cap of $125 billion. Facebook has a market cap of $216 billion. (source for cost: Cormorant Alpha shutdown in 2013) Edit: misplaced parenthesis => my first number was way off, $70 000 vs $100. Redid analysis, made mental note not to do math before coffee intake.
http://www.offshoreenergytoday.com/cormorant-alpha-shutdown-... estimates that the Cormorant Alpha shutdown cost $10,000,000/day, or about $100/second. That figure comes from multiplying the oil production (90,000 barrels/day) by the oil price ($110/barrel). Where did you get the $70,000/second figure from?
Gah, misplaced a parenthesis. You are correct. I shouldn't do math before I've had coffee.