Live data from Hacker News

Amazon EC2 and RDS in US-EAST zone down

status.aws.amazon.com

91–98 of 98 posts

Re: Amazon EC2 and RDS in US-EAST zone down

#91

Earlier quoted context omitted.

What zone? I really wish Amazon would provide that info, instead of saying that it only affects one zone.

Both my MySQL master (I'm not using RDS) and Redis Master servers are affected and are located in zone us-east-1a.

Why do you have two masters in the same AZ?

Re: Amazon EC2 and RDS in US-EAST zone down

#92
post #76

Earlier quoted context omitted.

Multi-AZ RDS deployment is supposed to protect you from that though. That's why it's 2x the price. We should have failed over to a different AZ w/o EBS issues.

If your source EBS volume is horked then you aren't going to be replicating any data to your backup host while the EBS volume is messed up (since your source data is unavailable). EBS volumes also don't cross/failover between AZ boundaries. Maybe there was something bad with your replication server before the outage? It's hard to guess without knowing exactly what was happening at the time...

I don't think you're familiar with how Multi AZ RDS works: http://aws.amazon.com/rds/faqs/#36

The whole point is to protect you from problems in one AZ by keeping a hot standby in another AZ. It doesn't matter whether it's due to EBS, power, etc. This is one of the primary reason to use RDS instead of running MySQL yourself on an instance.

Re: Amazon EC2 and RDS in US-EAST zone down

#93
post #53

Anyone have any details on why us-east-1 seems to be less reliable than the other regions? Is it the oldest?

According to this calculation (which attempted to probe all the racks in EC2), over 70% of EC2 lives in us-east.

http://huanliu.wordpress.com/2012/03/13/amazon-data-center-s...

Re: Amazon EC2 and RDS in US-EAST zone down

#94

Earlier quoted context omitted.

At $2300/month you could redundantly colo or lease VERY powerful servers in 3-4 data centers around the country.

Except when you have to factor in all the plane flights to replace broken HDD. And the risk of not making it in time for when it breaks.

Yeah, I don't think I'd go with less than RAID-6 (or full system redundancy plus 1 drive redundancy in each). Rebuilds just take too long, even with an in-chassis spare on RAID5.

Unfortunately Areca is really the only controller I've found which is well supported and does RAID6 fast.

Re: Amazon EC2 and RDS in US-EAST zone down

#95
post #92

Earlier quoted context omitted.

If your source EBS volume is horked then you aren't going to be replicating any data to your backup host while the EBS volume is messed up (since your source data is unavailable). EBS volumes also don't cross/failover between AZ boundaries. Maybe there was something bad with your replication server before the outage? It's hard to guess without knowing exactly what was happening at the time...

I don't think you're familiar with how Multi AZ RDS works: http://aws.amazon.com/rds/faqs/#36 The whole point is to protect you from problems in one AZ by keeping a hot standby in another AZ. It doesn't matter whether it's due to EBS, power, etc. This is one of the primary reason to use RDS instead of running MySQL yourself on an instance.

Yes...what also sounds plausible is that since this was an EBS outage that the underlying EBS volume wasn't detected as being unavailable (if it in fact did become unavilable) so no failover to your other RDS server was initiated.

Re: Amazon EC2 and RDS in US-EAST zone down

#97

Earlier quoted context omitted.

That sucks badly. Similar thing happened to me a while ago with a vendor. When your management team summons you to ask why the hell their site is down, you can't point fingers at the vendor if their marketing literature says it doesn't go down. Sticky situation.

If you don't host your data in several alternative dimensions so that the same events wouldn't transpire in all of them - why not assume you'll encounter the occasional outage?

If only people understood that fact. Unfortunately few do.

Re: Amazon EC2 and RDS in US-EAST zone down

#98

Earlier quoted context omitted.

I'm paying like 2,300 a month and even something basic like failover isnt working. I'm not happy.

At $2300/month you could redundantly colo or lease VERY powerful servers in 3-4 data centers around the country.

Poor inherited architecture, working to scale out greatnonprofits.org horizontally but it will be a while before we get there.

  I have nothing against colo but I don't really have time to run around the country checking on servers.
Post reply on HN