Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon RDS failure - data has been lost

news.ycombinator.com

Post reply on HN
Print view View original

41–48 of 48 posts

Re: Amazon RDS failure - data has been lost

#41

Post by acdha »

akhkharu wrote: post #9
acdha wrote: post #6

Earlier quoted context omitted.

> As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too. Which failure? The networking issue which had nothing to do with RDS and left your data unaffected? > Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data. Any se…

Yes, networking issue which brought down even multi-az deployments. I don't want to setup highly available fault tolerant systems, I just want a good level of reliability of a service provider. Probably, we will migrate to the dedicated servers out of Amazon soon. It will be harder to maintain, but cheaper and, as practice shows, more reliable.

> Yes, networking issue which brought down even multi-az deployments.

You might want to learn more about this before making business decisions on it. RDS was completely unaffected, as were all of my EC2 servers. They didn't receive any traffic from the internet but the systems were running fine throughout the brief outage interval.

A quick Google search will reveal that this is not uncommon for any hosting setup - data centers have lost network connections, routers can fail or be misconfigured, etc. - which is why anyone with serious uptime requirements has multiple widely separated data centers. Using AWS doesn't magically remove the need to avoid single points of failure in your system design.

> Probably, we will migrate to the dedicated servers out of Amazon soon. It will be harder to maintain, but cheaper and, as practice shows, more reliable.

I hope you have a good ops team and extra engineering resources; otherwise you'll learn very quickly that dedicated servers have the same failure modes. So far we're at ~18 minutes of AWS downtime this year - that's not going to be easy to beat.

Re: Amazon RDS failure - data has been lost

#42

Post by acdha »

nevinera wrote: post #5
akhkharu wrote: post #3

Earlier quoted context omitted.

As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too. Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data.

>Cloud solutions even with single AZ should not loss data. You mean you think all cloud db solutions should implement replication for you? There aren't very many backup solutions that never lose any data.

… and magically do it at no extra charge, too. Some learning experiences are in order.

Re: Amazon RDS failure - data has been lost

#43

Post by andrewvc »

danudey wrote: post #40
andrewvc wrote: post #37

Earlier quoted context omitted.

Yep, I've been telling this to people for years, and they keep sticking their fingers in their ears because they think they're entitled to magic uptime by amazon. The key to EBS is frequent S3 snapshots. The reliability issues of EBS only appears to the delta since your last S3 snapshot. Still... people use EBS w/o frequent S3 backups, amazingly.

Frequent snapshots only help you recover after a failure. RAID1 in software is a good way to prevent downtime, as well as potentially improve read speeds.

Not really, according to the EBS docs, they can pull missing blocks from EBS from S3. You shouldn't have to perform a restore. The uncertainty should only apply to the delta unless I'm mistaken.

Re: Amazon RDS failure - data has been lost

#44

Post by nevinera »

akhkharu wrote: post #11
nevinera wrote: post #5

Earlier quoted context omitted.

>Cloud solutions even with single AZ should not loss data. You mean you think all cloud db solutions should implement replication for you? There aren't very many backup solutions that never lose any data.

No, I understand that replication is the double cost. I meant cloud solution should not have storage failures which causes data loss.

And google should give us all ponies.

Would you mind defending your expectation that other people and companies will give you extra services for free?

Amazon has been very clear on the expected failure rate of ebs volumes and the attendant rds failures. If you want data safety, multi-az deployment offers it.

Re: Amazon RDS failure - data has been lost

#46

Post by bluestix »

semanticist wrote: post #39
EwanToo wrote: post #33

Earlier quoted context omitted.

The Netflix approach (which definitely isn't for everyone), is to use no persistent storage. They cluster their databases across multiple instances, availability zones and regions, and back them up constantly to S3. Their ultimate recovery plan is to restore from backup if necessary, which is pretty much the same as assuming EBS will corrupt your data. While it's far from trivial to do it that way, they seem to be th…

I'm more and more of the opinion that EC2 only makes sense for very small (2-3 node, no significant load) or very large (Netflix) deployments, and almost everyone else would be better off with either Linode-style VM rental or actual dedicated servers, depending on their needs. The only exception I've seen are people with very spiky traffic patterns, who can save a bit of money by only running instances when they need…

For any constant load applications dedicated servers are definitely much cheaper then ec2.

Amazon wins for APIs and convenience but not for price.

Re: Amazon RDS failure - data has been lost

#47

Post by JohnHaugeland »

electrum wrote: post #24
JohnHaugeland wrote: post #18

Earlier quoted context omitted.

They go to great effort to tell prospective customers that it's extremely reliable, providing claims of obscene numbers of nines. A real engineer should know better, but otherwise, it's people trusting what a major company claims. If it was a fly by night organization I would totally agree with you, but Amazon is a major multinational. It seems to me as reasonable for an outsider to trust the claims they make as it i…

Amazon only claims "obscene number of nines" for S3 durability (99.999999999%). And this claim seems to be accurate: I've never seen a publicly reported case of anyone losing data. Anytime you read their forums about people reporting data loss, a typical response is AWS staff saying "we see delete requests for those objects on date X" with the users responding "oh, oops, we had this background delete process". Howeve…

I withdraw my position as apparently in error.

Thank you for the exceptionally clear counterpoint.

Re: Amazon RDS failure - data has been lost

#48

Post by superuser2 »

akhkharu wrote: post #9
acdha wrote: post #6

Earlier quoted context omitted.

> As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too. Which failure? The networking issue which had nothing to do with RDS and left your data unaffected? > Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data. Any se…

Yes, networking issue which brought down even multi-az deployments. I don't want to setup highly available fault tolerant systems, I just want a good level of reliability of a service provider. Probably, we will migrate to the dedicated servers out of Amazon soon. It will be harder to maintain, but cheaper and, as practice shows, more reliable.

>It will be harder to maintain, but cheaper and, as practice shows, more reliable.

Practice shows that Amazon's uptime isn't perfect. Where's the evidence that a dedicated server is?

(There may well be evidence to support that. But I get awfully weary of the "it's not cloud so it must be more reliable" trap.)

Post reply on HN