Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon RDS failure - data has been lost

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 48 posts

Re: Amazon RDS failure - data has been lost

#11

Post by akhkharu »

nevinera wrote: post #5
akhkharu wrote: post #3

Earlier quoted context omitted.

As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too. Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data.

>Cloud solutions even with single AZ should not loss data. You mean you think all cloud db solutions should implement replication for you? There aren't very many backup solutions that never lose any data.

No, I understand that replication is the double cost.

I meant cloud solution should not have storage failures which causes data loss.

Re: Amazon RDS failure - data has been lost

#12

Post by EwanToo »

RDS should not have lost data, and if I were a user of it, I'd be annoyed too.

At the same time, if you've not spotted by now that EBS (elastic block storage, which powers RDS) is not reliable and not to be trusted, then you have to look at yourself too.

EBS is by far the worst product AWS offer, you simply should not use it without a very good reason, and if you do need to use it, you have to assume any given drive image will disappear at any moment - as it did here.

Beyond that, any time you're running a database, no matter who the provider is, if you're not doing backups every day or hour, then you're not doing things right.

Re: Amazon RDS failure - data has been lost

#14

Post by shiftpgdn »

PaulHoule wrote: post #10

If you had a database running on a dedi you could get trashed by a server failure too. Good backups are the best defense.

To me this is a fallacious argument. Dedicated servers are wildly cheaper than RDS/AWS. Isn't that the whole point of AWS? To have a team of experts managing your hosting to prevent a failure like this?

Re: Amazon RDS failure - data has been lost

#15

Post by PaulHoule »

shiftpgdn wrote: post #14
PaulHoule wrote: post #10

If you had a database running on a dedi you could get trashed by a server failure too. Good backups are the best defense.

To me this is a fallacious argument. Dedicated servers are wildly cheaper than RDS/AWS. Isn't that the whole point of AWS? To have a team of experts managing your hosting to prevent a failure like this?

I find it's very expensive to talk with salespeople to get my dedis configured properly. Particularly when they screw it up anyway.

The reason I moved to AWS was because when I added a new (cheap) hard drive to my dedi, they didn't put a partition table on the disk. When the machine rebooted, the superblock got overwritten and I lost access to the file system. (I did manage to recreate the superblock and get the data out, but jeese...)

One time I made a ticket and somehow my record in the trouble ticket system got screwed up and I couldn't put more tickets in. Some wizard fixed that in the SQL monitor after I talked to 3 other people who had no idea this could happen.

As for costs it's not so simple. I've got a processing job I run each week that costs $6 of CPU time because I pay just for what I need. I wouldnt want to run it on a dedi because it's a beast.

Re: Amazon RDS failure - data has been lost

#16

Post by philjohn »

akhkharu wrote: post #7
purephase wrote: post #4

I'm not sure I understand the "which does not have actual data" part of your statement. Could you explain that a bit more?

Point in Time backup was created before the actual failure and does not contain the latest data (~ 1 hour).

Surely you realised that is the case with a point-in-time backup? If you absolutely cannot lose data then as others have said, Multi AZ is required, or, at the very least, have a transaction log that you can replay (once again, hosted somewhere else).

Re: Amazon RDS failure - data has been lost

#17

Post by dspillett »

nevinera wrote: post #5
akhkharu wrote: post #3

Earlier quoted context omitted.

As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too. Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data.

>Cloud solutions even with single AZ should not loss data. You mean you think all cloud db solutions should implement replication for you? There aren't very many backup solutions that never lose any data.

A lot of people do (incorrectly) assume this sort of thing, even in cases when it would only take cursory understanding/research to spot the limitations a given service has and where it might fail in a disaster recovery situation.

No matter how good you think a given "cloud" solution is, no matter how much too big to fail you thing the company responsible is, you should make use of the redundancy options they provide and make sure you have reasonable backups elsewhere too (local to you or on another completely separate remote service).

This is what made me dismiss Google's App Engine when it first turned up (I've not looked for some time, they may have addressed this concern long ago): there was no easy way to backup all your data to another location/service and the not-so-easy ways would probably all end up costing a fair bit in bandwidth charges.

Re: Amazon RDS failure - data has been lost

#18

Post by JohnHaugeland »

EwanToo wrote: post #12

RDS should not have lost data, and if I were a user of it, I'd be annoyed too. At the same time, if you've not spotted by now that EBS (elastic block storage, which powers RDS) is not reliable and not to be trusted, then you have to look at yourself too. EBS is by far the worst product AWS offer, you simply should not use it without a very good reason, and if you do need to use it, you have to assume any given drive…

They go to great effort to tell prospective customers that it's extremely reliable, providing claims of obscene numbers of nines.

A real engineer should know better, but otherwise, it's people trusting what a major company claims.

If it was a fly by night organization I would totally agree with you, but Amazon is a major multinational. It seems to me as reasonable for an outsider to trust the claims they make as it is for me, a car industry outsider, to trust the claims that Chevrolet makes about my car.

Do you know how to properly quality evaluate everything in your life?

Hope you don't need a doctor, lawyer, or plumber soon.

Re: Amazon RDS failure - data has been lost

#19

Post by efsavage »

JohnHaugeland wrote: post #18
EwanToo wrote: post #12

RDS should not have lost data, and if I were a user of it, I'd be annoyed too. At the same time, if you've not spotted by now that EBS (elastic block storage, which powers RDS) is not reliable and not to be trusted, then you have to look at yourself too. EBS is by far the worst product AWS offer, you simply should not use it without a very good reason, and if you do need to use it, you have to assume any given drive…

They go to great effort to tell prospective customers that it's extremely reliable, providing claims of obscene numbers of nines. A real engineer should know better, but otherwise, it's people trusting what a major company claims. If it was a fly by night organization I would totally agree with you, but Amazon is a major multinational. It seems to me as reasonable for an outsider to trust the claims they make as it i…

I don't remember ever seeing "obscene numbers of nines" claimed by Amazon. For S3, yes, but not EBS. The '9' character doesn't even appear once on http://aws.amazon.com/ebs/.

What it does say is this:

"As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume.

Which to any engineer, "real" or otherwise, should be a pretty strong signal that something bad will happen.

Re: Amazon RDS failure - data has been lost

#20

Post by kennu »

EwanToo wrote: post #12

RDS should not have lost data, and if I were a user of it, I'd be annoyed too. At the same time, if you've not spotted by now that EBS (elastic block storage, which powers RDS) is not reliable and not to be trusted, then you have to look at yourself too. EBS is by far the worst product AWS offer, you simply should not use it without a very good reason, and if you do need to use it, you have to assume any given drive…

What would you recommend for persistent disk storage on AWS instead of EBS then? Assuming you need to put database files somewhere where they don't disappear when instances terminate, and your database can access them.
Post reply on HN