If they decide not to hide it, some interesting news is on its way ;)
Elaborate?
Twitter Completely Down
171–180 of 194 posts
Re: Twitter Completely Down
#172Earlier quoted context omitted.
Accidents happen. (expanding my reply) No risk assessment in the world will stop a cage monkey from tripping over a pile of 1Us and falling onto the big red button. Figure out what your pain threshold is and live with it.
I know of a chap, called Terry, who became known as Total Network Terry. TNT. Because he accidently plugged the wrong cable into the wrong jack in the wrong cable cupboard on the wrong floor. Just finding it took days to sort out, apparently.
That's like when I unplugged what I thought was our T1 line, but it turned out it was a neighbor's, who happened to be an ISP, which was a new business direction we were just moving into. And they didn't believe it was an accident...
Re: Twitter Completely Down
#173Earlier quoted context omitted.
This is more like a large manufacturer having a power outage. A fire would destroy machinery or stock. Nothing's getting lost here, something's just unavailable.
But a large manufacturer having a power outage, for, an hour like we are here, would cause a great deal of trouble with anything downstream of that manufacturer. And I'd be willing to bet that industry sources, mailing lists, forums, (whatever the manufacturing equivalent of HN is) would be discussing the outage, how bad it is, how this is going to completely screw up profits, how irritated they are, etc, etc.
You don't think GM or Ford have ever lost power for an hour? Did you ever read about it? Did the car dealerships suddenly run out of cars for one hour three weeks later? I think you are vastly overestimating the impact transient events like this have.
Re: Twitter Completely Down
#174"Today's turbulence explained" was just posted: http://blog.twitter.com/2012/06/todays-turbulence-explained....
Unfortunately there are no details, it just says "there was a cascading bug in one of our infrastructure components".
Re: Twitter Completely Down
#175Twitter is down. The very fabric of modern life is endangered
Re: Twitter Completely Down
#176I think this is another good example of how we as an industry are still unable to adequately assess risk properly. I'm fairly certain that the higher-ups in Twitter weren't told "We have pretty good failover protection, but there is a small risk of catastrophic failure where everything will go completely down." Whoever was in charge of disaster recovery obviously didn't really understand the risk. Just like the recen…
> ... and that there is an opportunity for "consultants" who understand this, kind of like security consultants?
Risk management consultants already exist! Many companies just choose to assess it internally, or the consultants themselves are inexperienced (often lacking practical experience).
Re: Twitter Completely Down
#177Earlier quoted context omitted.
They definitely lose money - their ad model is based on engagement, so if tweets aren't seen/can't be acted on, money is lost.
Adversely. Twitter being down causes a stir, which gets people talking even more. They quickly make up for the lost revenue as a result. There is no such thing as bad publicity.
This is going to lead to a loss of confidence amongst many - investors, advertisers, partners - who thought that Twitter was finally past its initial scaling/teething issues. It's not like there is anyone that hasn't heard of Twitter at this point in time. So yes, yes there can be bad publicity once something reaches the scale of twitter.
Re: Twitter Completely Down
#178Earlier quoted context omitted.
Adversely. Twitter being down causes a stir, which gets people talking even more. They quickly make up for the lost revenue as a result. There is no such thing as bad publicity.
This is going to lead to a loss of confidence amongst many - investors, advertisers, partners - who thought that Twitter was finally past its initial scaling/teething issues. It's not like there is anyone that hasn't heard of Twitter at this point in time. So yes, yes there can be bad publicity once something reaches the scale of twitter.
You mean once you switch away from Rails, you don't automatically scale?
Re: Twitter Completely Down
#179I think this is another good example of how we as an industry are still unable to adequately assess risk properly. I'm fairly certain that the higher-ups in Twitter weren't told "We have pretty good failover protection, but there is a small risk of catastrophic failure where everything will go completely down." Whoever was in charge of disaster recovery obviously didn't really understand the risk. Just like the recen…
It is simply a matter of perceived value and cost benefit. Why would a cio spend millions on Dr when the probability of diaster is so minute that the risk manager cannot even calculate it? Ok there is a risk that a plane will hit the pdc. .00000002%. And ultimately will our business grind to a halt? Or can we use a manual workaround until backups recover to sdc and we capture data lost since last backup. I mean I have a hard time taking this sort of risk seriously unless I'm running dialysis machines and someones life is at risk.
Re: Twitter Completely Down
#180x is down: http://blog.jgc.org/2010/10/x-is-down.html
This deserves to be the top comment. Your one liner nailed it. Twitter was down long enough for far more people not to notice than did notice. Shit goes down. It always will. Whining about how whoever needs backups or failover protection or distributed networks of servers across the planet or should use a VPS instead or a dedicated server instead or Heroku instead or EC2 instead or a combination of all that crap doesn't make you right. It makes you a speculator. No amount of fallbacks will give you 100% uptime ever. And calling this a massive failure is also ludicrous. It's just some downtime. It went right back up so chill.
These posts are so incredibly annoying. We can see if service x is down for ourselves. That isn't news. I could maybe accept these stories if the link on the front page was to a blog post stating that not only is service x down but why it went down for sure plus an added lesson we can learn from it. Short of that it's become an easy way for people to build up a trillion karma points. And if you want to tell me you don't care about karma then you're either lying or you have none. Enough with this crap. We'll find out ourselves but most of us won't actually because we have lives and by the time we go online to check our favorite wank-off site it'll probably be back up again like the past fifty times I've seen a story about Heroku/AWS/Twitter being down.