"Amazon's EBSs are a barrel of laughs in terms of performance and reliability" • Hacker News Board

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#71

Post by enjo » Fri, Mar 18, 2011, 5:33 PM UTC

andrewvc wrote: post #66

saurik wrote: post #52

Earlier quoted context omitted.

You have to snapshot at the system level anyway if you want a consistent snapshot: otherwise the filesystem (or your database) could have been reordering and delaying writes that end up not being part of the "consistent snapshot". This is simply not a RAID-specific issue, nor is it a problem with EBS (as it is generally easy to use LVM, xfs, and/or PostgreSQL to handle that part of the job).

True. However, for some cases where you don't mind losing some data due to a recovery process EBS snapshots are 'good enough'. Additionally, with a database like CouchDB with a 'crash only' design, it should work for some cases as well.

We use EBS snapshots as a last-resort backup. They're really convenient that way. We have a more robust backup system, but in the unlikely event that something goes wrong at least we have those snapshots, even if they're not perfect.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#72

Post by snorkel » Fri, Mar 18, 2011, 5:50 PM UTC

gruseom wrote: post #56

Anybody care to comment on using EC2 with local (what Amazon calls ephemeral) storage and backup to S3? Seems to me the advantages are: it's cheaper and you avoid the performance and reliability problems with EBS. The disadvantages?

Using EBS has other features that are hard to overlook, such as snapshots and ability to quickly move your volumes to another instance when an instance failure happens, or if you needed to change the size of an instance (which you couldn't do directly until very recently).

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#73

Post by tzs » Fri, Mar 18, 2011, 5:52 PM UTC

We've been looking at moving some or all of our stuff to either Amazon EC2/EBS/S3 or Rackspace cloud hosting, and it has been interesting.

Amazon seems more flexible, since you buy block storage (EBS) independent of instances. If you have an application that needs a massive amount of data, but only a little RAM and CPU, you can do it.

Rackspace, on the other hand, ties storage to instances. If you only need the RAM and CPU of the smallest instance (256 MB RAM) but need more than the 10 GB of disk space that provides, you need to go for a bigger instance, and so you'll probably end up with a bigger base price than at Amazon.

On the other hand, the storage at Rackspace is actual RAID storage directly attached to the machine you instance is on, so it is going to totally kick Amazon's butt for performance. Also, at Amazon you pay for I/O (something like $0.10 per million operations).

Looking at our existing main database and its usage, at Amazon we'd be paying more just for the I/O than we now pay for colo and bandwidth for the servers we own (not just the database servers...our whole setup!).

The big lesson we've taken away from our investigation so far as that Amazon is different from Rackspace, and both are different from running your own servers. Each of these three has a different set of capabilities and constraints, and so a solution designed for one will probably not work well if you just try to map it isomorphically to one of the others. You don't migrate to the cloud--you re-architect and rewrite to the cloud.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#74

Post by ceejayoz » Fri, Mar 18, 2011, 5:53 PM UTC

WALoeIII wrote: post #69

ceejayoz wrote: post #63

Earlier quoted context omitted.

The biggest issue I have with RDS is that I can't do a multi-master deployment to scale up writes. I've got a very write-heavy workload in my systems (roughly one write for every two reads).

You can't do multi-master with MySQL anyways, which until very recently has been the only "engine" RDS supports. Even if you could do multi-master, replication is still single threaded. You have to come up with your own sharding scheme. This is a limitation of MySQL not RDS.

You can't do multi-master with MySQL? News to me - we've been using circular replication between two servers, each a master and slave, for quite some time now.

Not possible with RDS, unfortunately, but works fine on two EC2 instances.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#75

Post by floodfx » Fri, Mar 18, 2011, 5:57 PM UTC

I'll probably be downvoted for this but seems to me the root cause of this problem is Reddit's architectural decision to remain in a single availability zone. If it wasn't EBS it could have been some other issue related to the single AZ that could have brought the site down. Blaming EBS, particularly if you knew it to be a potential weakness in your architecture, seems like a deflection of responsibility.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#76

Post by krakensden » Fri, Mar 18, 2011, 5:58 PM UTC

gruseom wrote: post #56

Anybody care to comment on using EC2 with local (what Amazon calls ephemeral) storage and backup to S3? Seems to me the advantages are: it's cheaper and you avoid the performance and reliability problems with EBS. The disadvantages?

All of your EC2 instances can disappear without warning and everything on the local storage is now gone forever.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#77

Post by snorkel » Fri, Mar 18, 2011, 6:04 PM UTC

gpapilion wrote: post #45

absconditus wrote: post #41

How is it that Amazon.com is so reliable if there are so many problems with their "cloud" products? Do they not use the same software to run their site?

If you understand the limitations of the various products you can build a VERY reliable service. The reddit assumption of a single datacenter and single technology to store that data was an engineering failure. They essentially didn't have a disaster recovery plan in place.

I'm sure reddit's engineers are as capable as any for producing a seemless disaster recovery plan, but the most common obstacle to implementing it is cost. Most web services choose the occasional risk of downtime in one data center instead of incurring the cost of being in two data centers at all times.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#78

Post by spidaman » Fri, Mar 18, 2011, 6:10 PM UTC

Kilimanjaro wrote: post #37

Lesson for startups: start in the cloud, grow your business, build your own cloud. Never trust critical parts of your business to others.

Netflix seems to be the biggest counter case - grew their data centers and effectively gave up and moved it to AWS. I suspect the sweet spot is doing a bit of public and private cloud, adjusting how much is on one or the other based on costs, service levels and capacity requirement volatility.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#79

Post by gruseom » Fri, Mar 18, 2011, 6:13 PM UTC

krakensden wrote: post #76

gruseom wrote: post #56

Anybody care to comment on using EC2 with local (what Amazon calls ephemeral) storage and backup to S3? Seems to me the advantages are: it's cheaper and you avoid the performance and reliability problems with EBS. The disadvantages?

All of your EC2 instances can disappear without warning and everything on the local storage is now gone forever.

That's the "backup to S3" part.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#80

Post by snorkel » Fri, Mar 18, 2011, 6:16 PM UTC

floodfx wrote: post #75

I'll probably be downvoted for this but seems to me the root cause of this problem is Reddit's architectural decision to remain in a single availability zone. If it wasn't EBS it could have been some other issue related to the single AZ that could have brought the site down. Blaming EBS, particularly if you knew it to be a potential weakness in your architecture, seems like a deflection of responsibility.

Perhaps reddit could've mitigated some downtime with some cross-zone redundancy, but the underlying frustration is that Amazon does not provide a well behaved storage solution, which is a very critical infrastructure component for most web services.