Maybe I'm being daft. Why does this sh*t never seem to affect www.amazon.com
Amazon EC2 currently down. Affecting Heroku, Reddit, Others
151–160 of 311 posts
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#152This would be a great time to post a guide to architecting systems for failover using AWS. Anyone got a great guide?
We (Netflix) have done a bunch of presentations on it which are on our slideshare page and across the internet. After this issue is over I can give a longer answer. In short, we've just evacuated the affected zone and are mostly recovered.
I'm guessing your Chaos Gorilla helped to harden your architecture against this threat.
Since you've mostly recovered, how did your system do? Are there side-cases that Chaos Gorilla didn't touch?
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#153Earlier quoted context omitted.
If it's systematically difficult to do it correctly, then the system is wrong.
. . . or the problem is inherently complex.
Surely true, but that's the purpose of a system in the first place: to manage complexity and make it predictable. You could argue that we have such a system in place, given how well the Internet works overall. The fact that this system has problems goes against what I believe is fully evident proof that such a system can, in fact, work even better.
We're not talking about a leap in order of magnitude of complexity here—just simple management of common human behavioral tendencies in order to promote more reliability. "The problem is inherently complex" is always true and will always be true, but it's no excuse for not designing a system to gracefully handle that complexity.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#154Heh, the cynical side of me would like to point out that this is a great way to get people to stop talking about wiping a user's kindle. :-)
This kind of thinking is poisonous. I know it's in good fun and it's fun to look for connections in things, but it is actually preposterous to think that Amazon would purposely disrupt wide swaths of highly paying customers for much of a day to bury one story about bad customer relations. My guess is there are a lot of people working very hard to try and solve this problem right now, let's not belittle their efforts…
It was a joke. I made the same joke earlier today. No-one is seriously going to believe this.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#155Earlier quoted context omitted.
Yet netflix is currently down, so maybe there is a problem with the chimp army?
The problem is likely the same as usual: if the damn control plane is down, it doesn't matter how robust your failover architecture is, because your requests to bring up new machines go unanswered. There's pretty much no way to architect around that one as an AWS user (apart from going fully multi-cloud, but "nobody" actually does that, at least at scale), and I'm kind of shocked that those bits of AWS are still not…
> apart from going fully multi-cloud, but "nobody" actually does that, at least at scale
Pirate Bay might disagree with that sentence: http://torrentfreak.com/pirate-bay-moves-to-the-cloud-become...
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#156Maybe I'm being daft. Why does this sh*t never seem to affect www.amazon.com
Amazon doesn't run their site off the same set of EC2 servers and systems that everyone else does. They just used their experience to build it.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#157Earlier quoted context omitted.
And Netflix's Simian Army is awesome: http://techblog.netflix.com/2011/07/netflix-simian-army.html They've even released the "Chaos Monkey" open source: http://techblog.netflix.com/2012/07/chaos-monkey-released-in...
Yet netflix is currently down, so maybe there is a problem with the chimp army?
Netflix not down for me here (Eastern US).
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#158Earlier quoted context omitted.
This kind of thinking is poisonous. I know it's in good fun and it's fun to look for connections in things, but it is actually preposterous to think that Amazon would purposely disrupt wide swaths of highly paying customers for much of a day to bury one story about bad customer relations. My guess is there are a lot of people working very hard to try and solve this problem right now, let's not belittle their efforts…
It was a joke. I made the same joke earlier today. No-one is seriously going to believe this.
actually, there's probably some few idiots who will believe it.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#159Heh, the cynical side of me would like to point out that this is a great way to get people to stop talking about wiping a user's kindle. :-)
This kind of thinking is poisonous. I know it's in good fun and it's fun to look for connections in things, but it is actually preposterous to think that Amazon would purposely disrupt wide swaths of highly paying customers for much of a day to bury one story about bad customer relations. My guess is there are a lot of people working very hard to try and solve this problem right now, let's not belittle their efforts…
[deleted]
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#160Dropbox runs on S3, they are unaffected? And Heroku's website seems okay. Only reddit is down. That's because of this?