Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon RDS failure - data has been lost

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 48 posts

Amazon RDS failure - data has been lost

#1

Post by akhkharu »

Our RDS instance is in "failure" state after 8 hours of downtime. Have to restore from point in time backup which does not have actual data.

Amazon says:

Jun 15, 4:03 AM PDT The RDS service is now operating normally. All affected Multi-AZ RDS instances operated normally throughout the power event after failing over. We were able to recover many Single AZ instances successfully, but storage volumes attached to some Single-AZ instances could not be restored, resulting in those instances being placed in Storage failure mode. Customers with automated backups turned on for an affected database instance have the option of initiating a Point-in-Time Restore operation. This will launch a new database instance using a backup of the affected database instance from before the event. To do this, follow these steps: 1) Log into the AWS Management console 2) Access the RDS tab, and select DB Instances on the left-side navigation 3) Select the affected database instance 4) Click on the "Restore to Point in Time" button 5) Select "Use Latest Restorable Time 6) Select a DB instance class that is at least the same size as the original DB instance 7) Make sure No Preference is selected for Availability Zone 8) Launch DB Instance and connect your application We will be following up here with the root cause of this event.

Re: Amazon RDS failure - data has been lost

#3

Post by akhkharu »

justincormack wrote: post #2

Use multi AZ then, which performed as expected. There have been so many warnings about single AZ that you would hope people get it by now.

As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too.

Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data.

Re: Amazon RDS failure - data has been lost

#5

Post by nevinera »

akhkharu wrote: post #3
justincormack wrote: post #2

Use multi AZ then, which performed as expected. There have been so many warnings about single AZ that you would hope people get it by now.

As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too. Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data.

>Cloud solutions even with single AZ should not loss data.

You mean you think all cloud db solutions should implement replication for you? There aren't very many backup solutions that never lose any data.

Re: Amazon RDS failure - data has been lost

#6

Post by acdha »

akhkharu wrote: post #3
justincormack wrote: post #2

Use multi AZ then, which performed as expected. There have been so many warnings about single AZ that you would hope people get it by now.

As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too. Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data.

> As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too.

Which failure? The networking issue which had nothing to do with RDS and left your data unaffected?

> Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data.

Any server can go down. The very modest increase for a multi-AZ setup buys you real, meaningful improvements as you just learned. I'm sorry that you had to learn a lesson the hard way but there's a reason why AWS recommends a multi-AZ deployment for failover and it's not revenue.

The next step up would require you having multiple widely separated servers, which is where you really start talking about large amounts of money because you're talking about non-trivial engineering and taking on the operational overhead of 24x7 support.

Re: Amazon RDS failure - data has been lost

#9

Post by akhkharu »

acdha wrote: post #6
akhkharu wrote: post #3

Earlier quoted context omitted.

As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too. Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data.

> As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too. Which failure? The networking issue which had nothing to do with RDS and left your data unaffected? > Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data. Any se…

Yes, networking issue which brought down even multi-az deployments.

I don't want to setup highly available fault tolerant systems, I just want a good level of reliability of a service provider. Probably, we will migrate to the dedicated servers out of Amazon soon. It will be harder to maintain, but cheaper and, as practice shows, more reliable.

Post reply on HN