Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon RDS failure - data has been lost

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 48 posts

Re: Amazon RDS failure - data has been lost

#21

Post by moe »

EwanToo wrote: post #12

RDS should not have lost data, and if I were a user of it, I'd be annoyed too. At the same time, if you've not spotted by now that EBS (elastic block storage, which powers RDS) is not reliable and not to be trusted, then you have to look at yourself too. EBS is by far the worst product AWS offer, you simply should not use it without a very good reason, and if you do need to use it, you have to assume any given drive…

you simply should not use it without a very good reason

EBS is fine as long as you're aware of the constraints and plan accordingly.

you have to assume any given drive image will disappear at any moment

Duh! You mean it behaves exactly as documented? That is outrageous.

Re: Amazon RDS failure - data has been lost

#22

Post by ceejayoz »

JohnHaugeland wrote: post #18
EwanToo wrote: post #12

RDS should not have lost data, and if I were a user of it, I'd be annoyed too. At the same time, if you've not spotted by now that EBS (elastic block storage, which powers RDS) is not reliable and not to be trusted, then you have to look at yourself too. EBS is by far the worst product AWS offer, you simply should not use it without a very good reason, and if you do need to use it, you have to assume any given drive…

They go to great effort to tell prospective customers that it's extremely reliable, providing claims of obscene numbers of nines. A real engineer should know better, but otherwise, it's people trusting what a major company claims. If it was a fly by night organization I would totally agree with you, but Amazon is a major multinational. It seems to me as reasonable for an outsider to trust the claims they make as it i…

> They go to great effort to tell prospective customers that it's extremely reliable, providing claims of obscene numbers of nines.

Where do they do this? All the docs I've seen are pretty clear that EC2/EBS stuff could disappear and that you have to plan a fault-tolerant system.

Re: Amazon RDS failure - data has been lost

#23

Post by ceejayoz »

akhkharu wrote: post #9
acdha wrote: post #6

Earlier quoted context omitted.

> As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too. Which failure? The networking issue which had nothing to do with RDS and left your data unaffected? > Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data. Any se…

Yes, networking issue which brought down even multi-az deployments. I don't want to setup highly available fault tolerant systems, I just want a good level of reliability of a service provider. Probably, we will migrate to the dedicated servers out of Amazon soon. It will be harder to maintain, but cheaper and, as practice shows, more reliable.

"Brought down" and "Brought down and lost data" are very different severities. Many businesses can handle occasional downtime as long as data's not disappearing into the ether.

Re: Amazon RDS failure - data has been lost

#24

Post by electrum »

JohnHaugeland wrote: post #18
EwanToo wrote: post #12

RDS should not have lost data, and if I were a user of it, I'd be annoyed too. At the same time, if you've not spotted by now that EBS (elastic block storage, which powers RDS) is not reliable and not to be trusted, then you have to look at yourself too. EBS is by far the worst product AWS offer, you simply should not use it without a very good reason, and if you do need to use it, you have to assume any given drive…

They go to great effort to tell prospective customers that it's extremely reliable, providing claims of obscene numbers of nines. A real engineer should know better, but otherwise, it's people trusting what a major company claims. If it was a fly by night organization I would totally agree with you, but Amazon is a major multinational. It seems to me as reasonable for an outsider to trust the claims they make as it i…

Amazon only claims "obscene number of nines" for S3 durability (99.999999999%). And this claim seems to be accurate: I've never seen a publicly reported case of anyone losing data. Anytime you read their forums about people reporting data loss, a typical response is AWS staff saying "we see delete requests for those objects on date X" with the users responding "oh, oops, we had this background delete process".

However, for EBS volumes, Amazon is very clear about the expected data loss rate:

"The durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot. As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that will typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives."

Re: Amazon RDS failure - data has been lost

#25

Post by electrum »

kennu wrote: post #20
EwanToo wrote: post #12

RDS should not have lost data, and if I were a user of it, I'd be annoyed too. At the same time, if you've not spotted by now that EBS (elastic block storage, which powers RDS) is not reliable and not to be trusted, then you have to look at yourself too. EBS is by far the worst product AWS offer, you simply should not use it without a very good reason, and if you do need to use it, you have to assume any given drive…

What would you recommend for persistent disk storage on AWS instead of EBS then? Assuming you need to put database files somewhere where they don't disappear when instances terminate, and your database can access them.

You use EBS and back it up using the built-in snapshot functionality, and/or back it up another way. RDS, which is presumably based on EBS, provides backups and snapshots for precisely this reason.

Re: Amazon RDS failure - data has been lost

#26

Post by seanp2k2 »

nevinera wrote: post #5
akhkharu wrote: post #3

Earlier quoted context omitted.

As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too. Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data.

>Cloud solutions even with single AZ should not loss data. You mean you think all cloud db solutions should implement replication for you? There aren't very many backup solutions that never lose any data.

This kinda highlights the problem with "cloud"...many people, even engineers, don't really understand what they're getting into. At least with a single server, you know what you're getting, and you have only yourself to blame if you didn't plan for a typical, known, documented failure mode.

Re: Amazon RDS failure - data has been lost

#27

Post by huggyface »

PaulHoule wrote: post #15
shiftpgdn wrote: post #14

Earlier quoted context omitted.

To me this is a fallacious argument. Dedicated servers are wildly cheaper than RDS/AWS. Isn't that the whole point of AWS? To have a team of experts managing your hosting to prevent a failure like this?

I find it's very expensive to talk with salespeople to get my dedis configured properly. Particularly when they screw it up anyway. The reason I moved to AWS was because when I added a new (cheap) hard drive to my dedi, they didn't put a partition table on the disk. When the machine rebooted, the superblock got overwritten and I lost access to the file system. (I did manage to recreate the superblock and get the data…

Your anecdote is interesting, but not relevant. The parent rightly observes that the whole point of a service like RDS is that you don't have to babysit it. If you still do then it's all of the disadvantages of your own box, plus more disadvantages.

Re: Amazon RDS failure - data has been lost

#28

Post by jahewson »

EwanToo wrote: post #12

RDS should not have lost data, and if I were a user of it, I'd be annoyed too. At the same time, if you've not spotted by now that EBS (elastic block storage, which powers RDS) is not reliable and not to be trusted, then you have to look at yourself too. EBS is by far the worst product AWS offer, you simply should not use it without a very good reason, and if you do need to use it, you have to assume any given drive…

You get what you pay for. Single-AZ databases were lost due to a failure in a single AZ, which Amazon tells you will happen. If you want durability you need Multi-AZ, which is the only place I'd put a production database.

Re: Amazon RDS failure - data has been lost

#30

Post by purephase »

huggyface wrote: post #27
PaulHoule wrote: post #15

Earlier quoted context omitted.

I find it's very expensive to talk with salespeople to get my dedis configured properly. Particularly when they screw it up anyway. The reason I moved to AWS was because when I added a new (cheap) hard drive to my dedi, they didn't put a partition table on the disk. When the machine rebooted, the superblock got overwritten and I lost access to the file system. (I did manage to recreate the superblock and get the data…

Your anecdote is interesting, but not relevant. The parent rightly observes that the whole point of a service like RDS is that you don't have to babysit it. If you still do then it's all of the disadvantages of your own box, plus more disadvantages.

Admittedly, I'm not that familiar with RDS, but I think there is a big difference between babysitting and common practices around data protection.

Always plan for failure regardless of where your data is, what claims are made, or how much you're paying.

Post reply on HN