Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon S3 Transfer Acceleration

aws.amazon.com

Post reply on HN
Print view View original

51–60 of 88 posts

Re: Amazon S3 Transfer Acceleration

#51

Post by chrisseaton »

mabbo wrote: post #43
extra88 wrote: post #29

Earlier quoted context omitted.

With Snowball, you don't need a huge pipe because your data is being sneaker-netted to one of their data centers. You don't need a ton of storage disks because they're lending Snowball to you in the short-term as a means of copying your data to S3 (and/or Glacier) for the long-term. If you can spare a 1 gigabit connection to saturate with S3 uploads, you can send 50TB in about two weeks. It takes about a week to requ…

Does it seriously take a week to get a snowball? You'd think with all the logistics Amazon owns...

I think they meant a week to request it, receive it, fill it, send it back, upload it to S3.

Re: Amazon S3 Transfer Acceleration

#52

Post by OniBait »

s9ix wrote: post #49

Did anyone else have some pretty terrible results from the speed tester? I was pretty optimistic seeing their sample screenshot - am I missing something? http://i.imgur.com/UwNBGf3.png

It says on the page: "Note: In general, the farther away you are from an Amazon S3 region, the higher the speed improvement you can expect from using Amazon S3 Transfer Acceleration. If you see similar speed results with and without the acceleration, your upload bandwidth or a system constraint might be limiting your speed."

So I suspect you might be having that problem in your test.

Re: Amazon S3 Transfer Acceleration

#53

Post by rsync »

nxzero wrote: post #4

What's the fastest & cheapest way to get data off Amazon's cloud services?

"What's the fastest & cheapest way to get data off Amazon's cloud services?"

We[1] may have a good answer for you ...

Let's assume you have terabytes of data, otherwise there's no difficulty, right ?

So, when you combine the HN readers discount pricing for 10TB datasets, which is 4c/GB/mo. with the fact that we support 's3cmd' in our environment:

  ssh user@rsync.net s3cmd get s3://rsync/mscdex.exe
... and the fact that we have no other charges (no transfer/usage/bandwidth charges) ...

... and the fact that we have 10gbps connectivity through he.net ...

It's possible that we would be a good fit.

You fire up a 10TB account for $400/mo and you issue s3cmd commands, over SSH, on your rsync.net account, which "pulls" the data from S3, at pretty much whatever speed Amazon can throw at us.[2]

Email us.

[1] rsync.net

[2] http://arstechnica.com/information-technology/2015/12/rsync-... ... scroll down to the part where they do the speed tests ...

Re: Amazon S3 Transfer Acceleration

#54

Post by extra88 »

mabbo wrote: post #43
extra88 wrote: post #29

Earlier quoted context omitted.

With Snowball, you don't need a huge pipe because your data is being sneaker-netted to one of their data centers. You don't need a ton of storage disks because they're lending Snowball to you in the short-term as a means of copying your data to S3 (and/or Glacier) for the long-term. If you can spare a 1 gigabit connection to saturate with S3 uploads, you can send 50TB in about two weeks. It takes about a week to requ…

Does it seriously take a week to get a snowball? You'd think with all the logistics Amazon owns...

A week to go from requesting a Snowball to having your data in the cloud. For example, place the request Monday morning, receive it Wednesday afternoon, immediately get started filling it, ship it back Thursday afternoon, they receive it on Saturday, hook it up, your data is in your AWS storage by the end of the day Sunday.

Re: Amazon S3 Transfer Acceleration

#55

Post by aroch »

s9ix wrote: post #49

Did anyone else have some pretty terrible results from the speed tester? I was pretty optimistic seeing their sample screenshot - am I missing something? http://i.imgur.com/UwNBGf3.png

If it makes you feel better, the accelerated ones are slower for me!

http://i.imgur.com/n6DhyHj.png

Re: Amazon S3 Transfer Acceleration

#57

Post by shortstuffsushi »

brianwawok wrote: post #26
shortstuffsushi wrote: post #19

Earlier quoted context omitted.

Even with that, I don't get it. "Moving lots of data either requires a huge pipe, or a ton of storage disks." With that, they offer their Snowball device, which, if I'm understanding correctly, holds up to 50TB (now 80TB), which they physically ship to you, and then you ship back to them. How does this fix either of the constraints (disk space / connection pipe)?

So you have 50TB of images to send me and a 10mbit internet pipe. Is it faster to FTP me the file or mail it on a hard drive?

To you and @nxzero, I'm not debating whether it's better or worse, I'm trying to find out if I'm actually understanding it right. From some of the other comments, it appears that it is just a matter of them sending you a physical disk -- it makes sense now.

Re: Amazon S3 Transfer Acceleration

#58

Post by hrez »

ghshephard wrote: post #20

Does anybody have a sense of how fast uploading to S3 can get? I'm on a gigabit link here in Singapore, uploading to Singapore instance of S3 (via ARQ) - and I'm disappointed that I rarely see better than about 40 Mbits/second.

try https://github.com/rlmcpherson/s3gof3r you can often get wire speed

Re: Amazon S3 Transfer Acceleration

#59

Post by lox »

ghshephard wrote: post #20

Does anybody have a sense of how fast uploading to S3 can get? I'm on a gigabit link here in Singapore, uploading to Singapore instance of S3 (via ARQ) - and I'm disappointed that I rarely see better than about 40 Mbits/second.

I wrote a benchmark a while back: https://github.com/lox/s3-bench

Re: Amazon S3 Transfer Acceleration

#60

Post by EwanToo »

dantiberian wrote: post #56

Do I understand this correctly that Cloudfront is still the better option for fronting static assets like JS and CSS?

Yes, cloudfront gives you far better control over things like headers, routing for different paths, error pages, etc, than vanilla s3 http access does, and is almost certainly lower latency compared to s3 (assuming your files aren't GB-sized)
Post reply on HN