AWS Storage Gateway • Hacker News Board

Re: AWS Storage Gateway

#31

Post by kisielk » Wed, Jan 25, 2012, 8:59 PM UTC

soult wrote: post #10

chubs wrote: post #8

Earlier quoted context omitted.

It sounds to me like enterprisey dropbox: It backs up your files from your local file server to the aws cloud, and if your server dies, bung in a new one and all your files will reappear. Great idea, although maybe something you could already do with dropbox?

As far as I understood the product page and Werner Vogels' blog post[1] it does not run on file level, but on filesystem level. So you won't be able to access single files from within S3, but rather have whole disk images stored in Amazon S3, ready to be restored back to your local datacenter or to be mounted on EC2 servers. 1: http://www.allthingsdistributed.com/2012/01/The-AWS-Storage-...

Actually even lower level than that. iSCSI just makes a block device available over a TCP/IP network. You can use that block device however you like, write random data to it, partition it in to multiple volumes, use it as part of a volume group or disk pool, etc. The individual block device doesn't need to have a filesystem on it.

Re: AWS Storage Gateway

#32

Post by tiernano » Wed, Jan 25, 2012, 9:05 PM UTC

sciurus wrote: post #26

moe wrote: post #20

Earlier quoted context omitted.

I'm not sure if that's what kondro meant but what I would like to see is a virtually unlimited size ("elastic") volume where the local disk acts only as the cache. This would make a wide range of big-storage use-cases ridiculously trivial - those where only ~10% of the data-set is frequently accessed. I.e. one could lazily scale the expensive local storage with throughput-demand, while the S3 backing store takes care…

Let's say you expect to grow to 20TB of data. Storing that for one month on S3 costs $2,560 (standard) or $1700 (reduced redundancy). In contrast, a Dell R515 with twelve 2TB drives costs $7,000. In a year that's one-third to one-quarter the price of using S3. Implementing a tiered storage system yourself is pretty complex. Using this S3 gateway might be simpler, but it's not trivial (e.g. you'll need VMware ESXi jus…

ok, when you get to the TB of storage part, things get cheaper running in house, but couple of notes:

1: the dell at $7k does not include power, and your 12 2Tb drives gives you 20Tb usuable with RAID6 (loosing 2 drives). if you loose more than 2 disks, you are screwed... so, you need to back that up somewhere... 2: you need someone to manage that machine also... 3: ESXi, for what you would need here (8gb ram or so) is free, unless you want support....

i think in all fairness, that depending on the amount of storage you need or want, its swings and round-a-bouts... i like the idea, but i would also like the idea of having a box in house with a lot of storage (like the big dell) and only select some parts for off site backup... this is what i do... most of my stuff is stored locally (RAID 1, Thecus NAS, Drobo) and only important stuff (music and videos i bought, photos i took, etc) is backed up to the "cloud"...

Re: AWS Storage Gateway

#33

Post by kondro » Wed, Jan 25, 2012, 9:30 PM UTC

sciurus wrote: post #24

kondro wrote: post #14

Earlier quoted context omitted.

Not since the last snapshot. It syncs as close to real-time as possible without the latency and network speeds affecting the performance locally. This solution isn't designed to replace fault-tolerance on local hardware. It is for close to realtime offsite replication and backup. Data in S3 is stored in at least three geographically separate locations and snapshots are very fast and very efficient on storage space. T…

Let's say I need 50TB of usable space. I can purchase an Equallogic PS6100E (a mid-level "enterprise" storage device) with twenty-four 3TB drives and 5 years of support for $85,000. Rack space and power isn't free, so let's say the total cost of equipment and facilities over 5 years is $100,000. In contrast, storing 50TB for 5 years on S3's reduced redundancy storage would cost $250,000. If I ever need to transfer an…

Except to get the same redundancy as this AWS product you would need a replica of this storage device in at least 2 other datacentres, you would have to ensure you have enough spare capacity lying around for any snapshots you take and have to buy or build software that manages this 3-way data synchronisation between sites.

Also, you seem to be forgetting that you still need local storage for your images anyway. This is a hands-off backup and disaster recovery product.

For full disclosure, the Storage Gateway's pricing isn't the same as S3's at the moment. They only have one storage tier and it is $0.14 per GB, no discounts. Therefore, 50TB of storage over 5 years is $420k.

Having said that, what would it cost to:

* Not only have your primary but 2 x secondary PS61000E's.

* 2 extra datacentres with connectivity to themselvse and your primary site.

* Software to manage asynchronous streaming of data from your primary to 2 x secondaries.

* Software to take consistent backups of these images and store in at all three locations.

* Software to ensure that your secondary sites contain only encrypted data.

* Cold-spare hardware at a secondary site capable of running all your images.

Re: AWS Storage Gateway

#34

Post by [deleted] » Wed, Jan 25, 2012, 9:30 PM UTC

sciurus wrote: post #24

kondro wrote: post #14

Earlier quoted context omitted.

Not since the last snapshot. It syncs as close to real-time as possible without the latency and network speeds affecting the performance locally. This solution isn't designed to replace fault-tolerance on local hardware. It is for close to realtime offsite replication and backup. Data in S3 is stored in at least three geographically separate locations and snapshots are very fast and very efficient on storage space. T…

Let's say I need 50TB of usable space. I can purchase an Equallogic PS6100E (a mid-level "enterprise" storage device) with twenty-four 3TB drives and 5 years of support for $85,000. Rack space and power isn't free, so let's say the total cost of equipment and facilities over 5 years is $100,000. In contrast, storing 50TB for 5 years on S3's reduced redundancy storage would cost $250,000. If I ever need to transfer an…

[deleted]