101–110 of 194 posts
I hope they write up a post-mortem on the fallout (hopefully it won't be a post-mortem of Twitter). Those things are always extremely interesting with big infrastructure like this.
https://mobile.twitter.com/ is working just fine.
Earlier quoted context omitted.
I agree that learning from why Twitter was down will be interesting and when that story comes out I hope it will be high on Hacker News. But this sort of news story about something that's happening right now is symptomatic of the useless '24 hour news' cycle of noise.
> But this sort of news story about something that's happening right now is symptomatic of the useless '24 hour news' cycle of noise. The single most important part of the internet is the immediate availability of news (to me, anyway). I've never heard anyone complain about that before; why do you think it's not worth knowing and talking about events as they happen? 'Twitter is down' isn't noise. Years and years ago…
Think about these two scenarios:
1. During that one hour you spend your time focussed on talking about this event as it's happening, speculating, having an emotional response (because you can't access something you want to and find a group of people experiencing the same thing and all get together to experience the frustration).
2. Tomorrow you read a story that says "Twitter was down for one hour yesterday" with some detail about what happened.
I believe that the latter is preferable. It's more efficient, less emotional and more useful. The former is the same as watching some 'Breaking News' event while is happening.
Now imagine that the one hour of downtime happened when you were asleep. You've missed nothing.
There are two scenarios where this news is important: if your business depends on Twitter, and if you are trying to assess the reliability of Twitter. The latter can be achieved by #2 above, only the former needs real-time updates and that doesn't mean general news reporting just your own monitoring.
Earlier quoted context omitted.
Well, it is newsworthy at least. Sure, being upset/getting angry just because of a little bit of Twitter downtime is stupid, but that doesn't take away from the fact that one of the biggest and most important discussion and communication channels the web has is completely down.
I don't believe it is newsworthy. In fact, it's sad that it is being written about because it shows the utter shallowness of what passes for 'Silicon Valley News' at the moment. And it's sad to see this as a 'top story' on Hacker News taking up space.
Think of "twitter being down" as Silicon Valleys equivalent to hollywoods "Lindsey Lohan is drunk in jail again"..
The tech companies, their founders staff and services are our pop-culture to gossip about.
I think this is another good example of how we as an industry are still unable to adequately assess risk properly. I'm fairly certain that the higher-ups in Twitter weren't told "We have pretty good failover protection, but there is a small risk of catastrophic failure where everything will go completely down." Whoever was in charge of disaster recovery obviously didn't really understand the risk. Just like the recen…
I'd guess the higher ups at Twitter have run the cost benefit in their head (and probably many spreadsheets) plenty of times, and in most cases spending your limited resource on disaster recovery preparation just isn't worth it. Their site being down does not qualify as a "disaster" - they'll be back up soon, then we'll all be tweeting away again within minutes.
The point is that at least in the case of Heroku and EC2 (I'm not sure what caused Twitter's outage yet), the causes of failures weren't something like a "16-sigma" event like a plane hitting an electrical tower (a tragic event that happened in Palo Alto a couple of years ago). They were things like insufficiently tested software and processes, and misconfigured devices. These things do not add cost, except maybe incrementally more man-hours in terms of testing and auditing of configurations. They are not million-dollar diesel units that require permits, etc.
My point is that if a device misconfiguration can take down EC2, or a single bad data in their data stream can cause massive failure, it means that the entire system is much more fragile than its been sold to everyone. If they didn't realize this was the case, it means that the risk of failure was a lot higher than they had assessed.
I hope they write up a post-mortem on the fallout (hopefully it won't be a post-mortem of Twitter). Those things are always extremely interesting with big infrastructure like this.
I think it's almost a requirement of these large platforms now. If they don't then developers would lose trust in them; I would anyway.
I think this is another good example of how we as an industry are still unable to adequately assess risk properly. I'm fairly certain that the higher-ups in Twitter weren't told "We have pretty good failover protection, but there is a small risk of catastrophic failure where everything will go completely down." Whoever was in charge of disaster recovery obviously didn't really understand the risk. Just like the recen…
It is likely that what you mean by "properly" is impossible. At large enough scales, what you end up with is a Gaussian distribution of errors in accordance with the Central Limit Theorem... except that there's a Black Swan spike in the low-probability, high-consequence events, and you basically can't spend enough money to ever get rid of them. Ever. Even if you try, you just end up piling equipment and people and procedures which will, themselves, create the black swan when they fail.
I think you're trying to imply that if only they'd understood better, this could absolutely have been prevented. No. Some specific action would probably have been able to avert this but you simply don't have a 100% chance of calling those actions in advance, no matter how good you are.
The state space of these systems is incomprehensibly enormous and there is no feasible way in which you can get all the failures out of it, neither in theory nor in practice.
Living in terror of the absolute certainty of eventual failure is left as an exercise for the reader.
I think this is another good example of how we as an industry are still unable to adequately assess risk properly. I'm fairly certain that the higher-ups in Twitter weren't told "We have pretty good failover protection, but there is a small risk of catastrophic failure where everything will go completely down." Whoever was in charge of disaster recovery obviously didn't really understand the risk. Just like the recen…