Adding insult to injury: when trying to view instances in the console I now only see "Rate limit exceeded".
Amazon EC2 currently down. Affecting Heroku, Reddit, Others
141–150 of 311 posts
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#142As https://twitter.com/DEVOPS_BORAT says, At conference you can able tell cloud devops by they are always leave dinner for respond to pager. Also, What is happen in cloud is stay in cloud because nobody can able reproduce outside of cloud. (And many other relevant quotes.)
Possibly the most relevant:
"Source of Amazon is tell me best monitoring strategy is watch Netflix. If is up, they can able blame customer. If is down, they are fuck."
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#143Why the fuck do the two most critical services (ELB and Console) have depends on their historically most unreliable pile of shit (EBS)?
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#144Isn't it worrying that there are so many services/sites that are so dependent on Amazon EC?
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#145As https://twitter.com/DEVOPS_BORAT says, At conference you can able tell cloud devops by they are always leave dinner for respond to pager. Also, What is happen in cloud is stay in cloud because nobody can able reproduce outside of cloud. (And many other relevant quotes.)
To be fair, AWS downtime always make the news because they affect a lot of majors websites, but that doesn't mean an average sysadmin (or devops, whatever) would do better in term of uptime with his own bay and his toys.
But this is part of the problem: we have multiple web properties, and the fact that AWS issues can affect all of them at once is a huge downside. Certainly, if we ran on metal, we would have hardware fail, but failures would be likely to be better-isolated than at Amazon.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#146Woot, Back Online www.zenwheels.com
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#147The N. Virginia datacenter has been historically unreliable. I moved my personal projects to the West Coast (Oregon and N. California) and I have seen no significant issues in the past year. N. Virginia is both cheaper and closer to the center of mass of the developed world. I'm surprised Amazon hasn't managed to make it more reliable.
Their CoLo space is the same space shared by AOL and a few other big name tech companies. It's right next to the Greenway, just before you reach IAD going northeast. That CoLo facility seems pretty unreliable in the scheme of things; Verizon and Amazon both took major downtime this summer when a pretty hefty storm rolled through VA[1], but AOL's dedicated datacenters in the same 10 mile radius all experienced no downtime whatsoever.
Edited: [1] http://www.datacenterknowledge.com/archives/2012/06/30/amazo...
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#148Heh, the cynical side of me would like to point out that this is a great way to get people to stop talking about wiping a user's kindle. :-)
This kind of thinking is poisonous. I know it's in good fun and it's fun to look for connections in things, but it is actually preposterous to think that Amazon would purposely disrupt wide swaths of highly paying customers for much of a day to bury one story about bad customer relations. My guess is there are a lot of people working very hard to try and solve this problem right now, let's not belittle their efforts because of a conspiracy, let's belittle their efforts because of bad systems design.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#149The N. Virginia datacenter has been historically unreliable. I moved my personal projects to the West Coast (Oregon and N. California) and I have seen no significant issues in the past year. N. Virginia is both cheaper and closer to the center of mass of the developed world. I'm surprised Amazon hasn't managed to make it more reliable.
I'm surprised amazon hasn't built another region in the east. If you're in the west you get US-West-1 and US-West-2 and can failover and distribute between the two, why don't they have that kind of duplication in the east?
Stop thinking about regions as datacenters.
us-east-1 was 11 different datacenters last time I bothered to check.
us-west-2 by comparison is two datacenters. The reason west-1 and west-2 exist is because they are geographically diverse enough to prevent low latency inner-connections (and also have dramatically different power costs so they bill differently).
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#150This would be a great time to post a guide to architecting systems for failover using AWS. Anyone got a great guide?
PagerDuty's CTO gave a talk that mentions it: http://blog.pagerduty.com/2012/10/ensuring-the-call-goes-out...