Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Storage Gateway

aws.amazon.com

Post reply on HN
Print view View original

21–30 of 34 posts

Re: AWS Storage Gateway

#21

Post by plasma »

moe wrote: post #20
tiernano wrote: post #16

Earlier quoted context omitted.

the volumes are cached... from reading the post, you write synchronously to the iSCSI device, which is Asynchronously sent to S3

I'm not sure if that's what kondro meant but what I would like to see is a virtually unlimited size ("elastic") volume where the local disk acts only as the cache. This would make a wide range of big-storage use-cases ridiculously trivial - those where only ~10% of the data-set is frequently accessed. I.e. one could lazily scale the expensive local storage with throughput-demand, while the S3 backing store takes care…

He's referring to the upcoming feature in the second paragraph at http://aws.amazon.com/storagegateway/faqs/#How_Storage_Gatew...

Re: AWS Storage Gateway

#22

Post by SomeOtherGuy »

This is exactly what we need. Except that the pricing on S3 storage is just too high. It is way more expensive than just buying our own replicating storage and hosting one off site. 25TB in s3 is is $43,000/year. It cost us $90,000 to get 4 falconstore ga700s and (3x40TB) of storage. This gives us 2 devices on site in a HA setup with 40TB of storage, and 2 offsite devices in different datacenters with 40TB of storage each. We end up with way more features than amazon's offering, better performance, and the 40TB of storage we have costs less than 3 years of having 25TB of storage at amazon, where we'd still need to be paying for the local storage as well.

Re: AWS Storage Gateway

#23

Post by nburger »

donky_cong wrote: post #2

So you are basically paying 125$+ to have S3 cached locally and get an iSCSI interface ?

Note that the $125 "gateway" fee is minor compared with the storage/transfer fees themselves - $0.14/GB/month for storage, for example. At that rate, a small 5 TB volume is already $700/month.

Re: AWS Storage Gateway

#24

Post by sciurus »

kondro wrote: post #14
cperciva wrote: post #5

Earlier quoted context omitted.

Note that the mirroring is asynchronous -- if your local server fails you can replace it with an EC2 instance, but you lose anything since the last snapshot.

Not since the last snapshot. It syncs as close to real-time as possible without the latency and network speeds affecting the performance locally. This solution isn't designed to replace fault-tolerance on local hardware. It is for close to realtime offsite replication and backup. Data in S3 is stored in at least three geographically separate locations and snapshots are very fast and very efficient on storage space. T…

Let's say I need 50TB of usable space. I can purchase an Equallogic PS6100E (a mid-level "enterprise" storage device) with twenty-four 3TB drives and 5 years of support for $85,000. Rack space and power isn't free, so let's say the total cost of equipment and facilities over 5 years is $100,000.

In contrast, storing 50TB for 5 years on S3's reduced redundancy storage would cost $250,000. If I ever need to transfer any of that data back to my data center, there'll be a hefty bill for that as well.

Re: AWS Storage Gateway

#25

Post by bwarp »

This is really cool. Having just done a large and complicated S3 integration into a legacy soup of 20 year old filesystem based document management kludge, this would have made life much easier (and considerably cheaper!).

Not only that, it solves a lot of problems such as dangerously storing backup snapshots on-site, archival and easy deployment and access to S3's CDN functionality.

Sold as far as I am concerned!

(Yes I know it's expensive, but it's cheaper than buying something in-house and employing another hairless monkey to manage it).

Re: AWS Storage Gateway

#26

Post by sciurus »

moe wrote: post #20
tiernano wrote: post #16

Earlier quoted context omitted.

the volumes are cached... from reading the post, you write synchronously to the iSCSI device, which is Asynchronously sent to S3

I'm not sure if that's what kondro meant but what I would like to see is a virtually unlimited size ("elastic") volume where the local disk acts only as the cache. This would make a wide range of big-storage use-cases ridiculously trivial - those where only ~10% of the data-set is frequently accessed. I.e. one could lazily scale the expensive local storage with throughput-demand, while the S3 backing store takes care…

Let's say you expect to grow to 20TB of data. Storing that for one month on S3 costs $2,560 (standard) or $1700 (reduced redundancy). In contrast, a Dell R515 with twelve 2TB drives costs $7,000. In a year that's one-third to one-quarter the price of using S3.

Implementing a tiered storage system yourself is pretty complex. Using this S3 gateway might be simpler, but it's not trivial (e.g. you'll need VMware ESXi just to get started).

Re: AWS Storage Gateway

#27

Post by evandena »

sciurus wrote: post #24
kondro wrote: post #14

Earlier quoted context omitted.

Not since the last snapshot. It syncs as close to real-time as possible without the latency and network speeds affecting the performance locally. This solution isn't designed to replace fault-tolerance on local hardware. It is for close to realtime offsite replication and backup. Data in S3 is stored in at least three geographically separate locations and snapshots are very fast and very efficient on storage space. T…

Let's say I need 50TB of usable space. I can purchase an Equallogic PS6100E (a mid-level "enterprise" storage device) with twenty-four 3TB drives and 5 years of support for $85,000. Rack space and power isn't free, so let's say the total cost of equipment and facilities over 5 years is $100,000. In contrast, storing 50TB for 5 years on S3's reduced redundancy storage would cost $250,000. If I ever need to transfer an…

With Amazon you could facilitate a DR onto their other platforms. With your Equallogic example you would need to mirror that data to another device with recovery targets in another location.

Re: AWS Storage Gateway

#29

Post by moe »

sciurus wrote: post #26
moe wrote: post #20

Earlier quoted context omitted.

I'm not sure if that's what kondro meant but what I would like to see is a virtually unlimited size ("elastic") volume where the local disk acts only as the cache. This would make a wide range of big-storage use-cases ridiculously trivial - those where only ~10% of the data-set is frequently accessed. I.e. one could lazily scale the expensive local storage with throughput-demand, while the S3 backing store takes care…

Let's say you expect to grow to 20TB of data. Storing that for one month on S3 costs $2,560 (standard) or $1700 (reduced redundancy). In contrast, a Dell R515 with twelve 2TB drives costs $7,000. In a year that's one-third to one-quarter the price of using S3. Implementing a tiered storage system yourself is pretty complex. Using this S3 gateway might be simpler, but it's not trivial (e.g. you'll need VMware ESXi jus…

Well, I only glanced at their current offering, missed the VMware part. My request was mostly wishful thinking.

I.e. instead of VMware it'd be more useful for us to hook in with a FUSE-layer or a patched variant of a filesystem such as GlusterFS.

You're of course correct about the pricing. Their current prices cover some middle-ground but would need to be discounted to make it feasible for larger deployments. However, at the low-end (your 20T figure) the price seems already justifiable when you factor in staff and infrastructure costs (rack+power alone make up for half of the difference).

Re: AWS Storage Gateway

#30

Post by nknight »

evandena wrote: post #27
sciurus wrote: post #24

Earlier quoted context omitted.

Let's say I need 50TB of usable space. I can purchase an Equallogic PS6100E (a mid-level "enterprise" storage device) with twenty-four 3TB drives and 5 years of support for $85,000. Rack space and power isn't free, so let's say the total cost of equipment and facilities over 5 years is $100,000. In contrast, storing 50TB for 5 years on S3's reduced redundancy storage would cost $250,000. If I ever need to transfer an…

With Amazon you could facilitate a DR onto their other platforms. With your Equallogic example you would need to mirror that data to another device with recovery targets in another location.

At the scales he's talking about, 10gbps uplinks are easily available. That said, the killer isn't the equipment costs, it's the people.
Post reply on HN