Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon RDS failure - data has been lost

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 48 posts

Re: Amazon RDS failure - data has been lost

#31

Post by kiallmacinnes »

akhkharu wrote: post #11
nevinera wrote: post #5

Earlier quoted context omitted.

>Cloud solutions even with single AZ should not loss data. You mean you think all cloud db solutions should implement replication for you? There aren't very many backup solutions that never lose any data.

No, I understand that replication is the double cost. I meant cloud solution should not have storage failures which causes data loss.

I disagree. Amazon simply gave you the choice over what level of reliability you require.

Would you have preferred they only offered Multi-AZ databases? (and in the process doubled your development, staging and QA environment costs..)

Re: Amazon RDS failure - data has been lost

#32

Post by snorkel »

EwanToo wrote: post #12

RDS should not have lost data, and if I were a user of it, I'd be annoyed too. At the same time, if you've not spotted by now that EBS (elastic block storage, which powers RDS) is not reliable and not to be trusted, then you have to look at yourself too. EBS is by far the worst product AWS offer, you simply should not use it without a very good reason, and if you do need to use it, you have to assume any given drive…

You can snapshot EBS volumes, however your EBS disk I/O will be blocked while the snapshot is being made, so you can't do it often on a large busy volume.

You can also raid EBS volumes together for realtime redundancy, but adds cost and complexity.

Re: Amazon RDS failure - data has been lost

#33

Post by EwanToo »

kennu wrote: post #20
EwanToo wrote: post #12

RDS should not have lost data, and if I were a user of it, I'd be annoyed too. At the same time, if you've not spotted by now that EBS (elastic block storage, which powers RDS) is not reliable and not to be trusted, then you have to look at yourself too. EBS is by far the worst product AWS offer, you simply should not use it without a very good reason, and if you do need to use it, you have to assume any given drive…

What would you recommend for persistent disk storage on AWS instead of EBS then? Assuming you need to put database files somewhere where they don't disappear when instances terminate, and your database can access them.

The Netflix approach (which definitely isn't for everyone), is to use no persistent storage.

They cluster their databases across multiple instances, availability zones and regions, and back them up constantly to S3. Their ultimate recovery plan is to restore from backup if necessary, which is pretty much the same as assuming EBS will corrupt your data.

While it's far from trivial to do it that way, they seem to be the most successful user of the AWS systems.

Re: Amazon RDS failure - data has been lost

#34

Post by snorkel »

justincormack wrote: post #2

Use multi AZ then, which performed as expected. There have been so many warnings about single AZ that you would hope people get it by now.

Sure, just duplicate your entire stack in at least two zones and replicate all data in realtime, and then you just need to convince your boss/investors/yourself that spending 2X on hosting is worth it. Once you add up the costs you soon realize that risking several hours of downtime once per year is more acceptable than doubling hosting costs. For anyone who is not hosting air traffic control or banking systems it's really not worth it. Seriously, if your web service is hosting social brain farts or selling cup holders, and not landing a space shuttles, then it can be offline a for few hours per year.

Re: Amazon RDS failure - data has been lost

#35

Post by count »

huggyface wrote: post #27
PaulHoule wrote: post #15

Earlier quoted context omitted.

I find it's very expensive to talk with salespeople to get my dedis configured properly. Particularly when they screw it up anyway. The reason I moved to AWS was because when I added a new (cheap) hard drive to my dedi, they didn't put a partition table on the disk. When the machine rebooted, the superblock got overwritten and I lost access to the file system. (I did manage to recreate the superblock and get the data…

Your anecdote is interesting, but not relevant. The parent rightly observes that the whole point of a service like RDS is that you don't have to babysit it. If you still do then it's all of the disadvantages of your own box, plus more disadvantages.

You don't though, as noted above. If you checked the 'multi-az' check box, which costs a little extra, then everything worked properly.

If you cheaped out and used a single-AZ deployment, against published best practices, than it's your own damn fault.

Re: Amazon RDS failure - data has been lost

#36

Post by EwanToo »

snorkel wrote: post #32
EwanToo wrote: post #12

RDS should not have lost data, and if I were a user of it, I'd be annoyed too. At the same time, if you've not spotted by now that EBS (elastic block storage, which powers RDS) is not reliable and not to be trusted, then you have to look at yourself too. EBS is by far the worst product AWS offer, you simply should not use it without a very good reason, and if you do need to use it, you have to assume any given drive…

You can snapshot EBS volumes, however your EBS disk I/O will be blocked while the snapshot is being made, so you can't do it often on a large busy volume. You can also raid EBS volumes together for realtime redundancy, but adds cost and complexity.

True, you're probably better off running the replication at the database layer, which can normally by either synchronous or asynchronous depending on what people need.

For people who don't want to spend the money on a hot standby, you can still produce the database log files from MySQL (or whatever you're using), and copy them off the machine onto S3 every few seconds, minutes, etc.

There's no end of solutions to the problem but, for whatever reason, a lot of people don't see EBS as a potential problem area.

Re: Amazon RDS failure - data has been lost

#37

Post by andrewvc »

efsavage wrote: post #19
JohnHaugeland wrote: post #18

Earlier quoted context omitted.

They go to great effort to tell prospective customers that it's extremely reliable, providing claims of obscene numbers of nines. A real engineer should know better, but otherwise, it's people trusting what a major company claims. If it was a fly by night organization I would totally agree with you, but Amazon is a major multinational. It seems to me as reasonable for an outsider to trust the claims they make as it i…

I don't remember ever seeing "obscene numbers of nines" claimed by Amazon. For S3, yes, but not EBS. The '9' character doesn't even appear once on http://aws.amazon.com/ebs/ . What it does say is this: "As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete lo…

Yep, I've been telling this to people for years, and they keep sticking their fingers in their ears because they think they're entitled to magic uptime by amazon.

The key to EBS is frequent S3 snapshots. The reliability issues of EBS only appears to the delta since your last S3 snapshot. Still... people use EBS w/o frequent S3 backups, amazingly.

Re: Amazon RDS failure - data has been lost

#38

Post by mschalle »

Always assume Murphy's law will hold, regardless of what service provider you use.

If you were running your own database, you surely would have had rigorous backups because the responsibility was on you.

Assume that if a service can fail, it will. If data can be lost, it will be. Then, plan accordingly.

EDIT: grammar

Re: Amazon RDS failure - data has been lost

#39

Post by semanticist »

EwanToo wrote: post #33
kennu wrote: post #20

Earlier quoted context omitted.

What would you recommend for persistent disk storage on AWS instead of EBS then? Assuming you need to put database files somewhere where they don't disappear when instances terminate, and your database can access them.

The Netflix approach (which definitely isn't for everyone), is to use no persistent storage. They cluster their databases across multiple instances, availability zones and regions, and back them up constantly to S3. Their ultimate recovery plan is to restore from backup if necessary, which is pretty much the same as assuming EBS will corrupt your data. While it's far from trivial to do it that way, they seem to be th…

I'm more and more of the opinion that EC2 only makes sense for very small (2-3 node, no significant load) or very large (Netflix) deployments, and almost everyone else would be better off with either Linode-style VM rental or actual dedicated servers, depending on their needs.

The only exception I've seen are people with very spiky traffic patterns, who can save a bit of money by only running instances when they need them - but even then I suspect that the money saved might be less than you'd think.

Re: Amazon RDS failure - data has been lost

#40

Post by danudey »

andrewvc wrote: post #37
efsavage wrote: post #19

Earlier quoted context omitted.

I don't remember ever seeing "obscene numbers of nines" claimed by Amazon. For S3, yes, but not EBS. The '9' character doesn't even appear once on http://aws.amazon.com/ebs/ . What it does say is this: "As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete lo…

Yep, I've been telling this to people for years, and they keep sticking their fingers in their ears because they think they're entitled to magic uptime by amazon. The key to EBS is frequent S3 snapshots. The reliability issues of EBS only appears to the delta since your last S3 snapshot. Still... people use EBS w/o frequent S3 backups, amazingly.

Frequent snapshots only help you recover after a failure. RAID1 in software is a good way to prevent downtime, as well as potentially improve read speeds.
Post reply on HN