Amazon S3 Transfer Acceleration • Hacker News Board

Re: Amazon S3 Transfer Acceleration

#11

Post by ladon86 » Tue, Apr 19, 2016, 5:11 PM UTC

toomuchtodo wrote: post #2

TL;DR Newer, larger "snowball" appliance (current = 50TB and newer 80TB version), as well as access in additional international regions. Network optimized S3 uploads (faster inbound transfers) for an extra 4 cents/GB. Sidenote: I'm curious if you can get the same speedup for inbound transfers by using a Cloudfront distribution in front of your S3 bucket for no additional charge (no cost for inbound transfer on either…

You can't upload through Cloudfront. It's a good analogy though, you can think of this feature as a "CDN for uploads", if you'll excuse the abuse of the CDN acronym.

Re: Amazon S3 Transfer Acceleration

#12

Post by narfz » Tue, Apr 19, 2016, 5:12 PM UTC

is there a way to run the benchmark from CLI? i would like to run the benchmark on our servers to see if we can get an improvement.

Re: Amazon S3 Transfer Acceleration

#13

Post by toomuchtodo » Tue, Apr 19, 2016, 5:13 PM UTC

ladon86 wrote: post #11

toomuchtodo wrote: post #2

TL;DR Newer, larger "snowball" appliance (current = 50TB and newer 80TB version), as well as access in additional international regions. Network optimized S3 uploads (faster inbound transfers) for an extra 4 cents/GB. Sidenote: I'm curious if you can get the same speedup for inbound transfers by using a Cloudfront distribution in front of your S3 bucket for no additional charge (no cost for inbound transfer on either…

You can't upload through Cloudfront. It's a good analogy though, you can think of this feature as a "CDN for uploads", if you'll excuse the abuse of the CDN acronym.

You cant? PUT/POST through Cloudfront supported since October 2013.

https://aws.amazon.com/about-aws/whats-new/2013/10/15/amazon...

"You can use existing Amazon CloudFront distributions for upload requests by simply enabling support in the AWS management console. When end users upload content, CloudFront will send the upload request back to the origin web server (such as an Amazon S3 bucket, an Amazon EC2 instance, an Elastic Load Balancer, or your own origin server) over an optimized route that uses persistent connections, TCP/IP and network path optimizations."

Re: Amazon S3 Transfer Acceleration

#14

Post by cmwright » Tue, Apr 19, 2016, 5:13 PM UTC

toomuchtodo wrote: post #2

TL;DR Newer, larger "snowball" appliance (current = 50TB and newer 80TB version), as well as access in additional international regions. Network optimized S3 uploads (faster inbound transfers) for an extra 4 cents/GB. Sidenote: I'm curious if you can get the same speedup for inbound transfers by using a Cloudfront distribution in front of your S3 bucket for no additional charge (no cost for inbound transfer on either…

We played with this a bit using both cloudfront and other cdns. While we did see some small improvements, it wasn't nearly as good as we were expecting on the upload side. It may be fixed now, but there were also many issues getting the proper headers to pass through cloudfront to s3 for chunked uploads.

Re: Amazon S3 Transfer Acceleration

#15

Post by jewel » Tue, Apr 19, 2016, 5:14 PM UTC

nxzero wrote: post #4

What's the fastest & cheapest way to get data off Amazon's cloud services?

I know of three ways to get data out of amazon's cloud. Whether they are the fastest depends on how much data you're talking about.

1) Download the data and pay egress bandwidth charges. Starts at $90/TB, but gets cheaper with more usage.

2) Use import/export snowball to have disks shipped to you. This is $30/TB + $250 per 80TB device.

3) Use direct connect to connect fiber directly to the region. Costs $1620/mo for each 10G line + $30/TB + an unknown amount to your fiber provider.

It's too bad that export traffic can't be marked as "low priority" with a cheaper cost. I imagine there are times at night when utilization is low and big export jobs could be run during those times. (It's obviously not in AWS's interest to make it cheap to get data out of their data centers.)

Re: Amazon S3 Transfer Acceleration

#16

Post by nxzero » Tue, Apr 19, 2016, 5:14 PM UTC

morgante wrote: post #6

nxzero wrote: post #4

What's the fastest & cheapest way to get data off Amazon's cloud services?

It depends entirely on the quantity of data. For huge datasets, Snowball will be fastest & cheapest (literally shipping data).

Great, thanks, here's the data plan to grab:

https://aws.amazon.com/datasets/

For downloading & uploading very large datasets, would it make more sense to proxy the through Amazon to a Snowball and ship it. Very possible is the answer is it depends, just trying to get a sense of how data transfer via Amazon is works and is priced.

Re: Amazon S3 Transfer Acceleration

#17

Post by [deleted] » Tue, Apr 19, 2016, 5:18 PM UTC

cmwright wrote: post #14

toomuchtodo wrote: post #2

TL;DR Newer, larger "snowball" appliance (current = 50TB and newer 80TB version), as well as access in additional international regions. Network optimized S3 uploads (faster inbound transfers) for an extra 4 cents/GB. Sidenote: I'm curious if you can get the same speedup for inbound transfers by using a Cloudfront distribution in front of your S3 bucket for no additional charge (no cost for inbound transfer on either…

We played with this a bit using both cloudfront and other cdns. While we did see some small improvements, it wasn't nearly as good as we were expecting on the upload side. It may be fixed now, but there were also many issues getting the proper headers to pass through cloudfront to s3 for chunked uploads.

[deleted]

Re: Amazon S3 Transfer Acceleration

#18

Post by GeechieCloud » Tue, Apr 19, 2016, 5:27 PM UTC

nxzero wrote: post #16

morgante wrote: post #6

Earlier quoted context omitted.

It depends entirely on the quantity of data. For huge datasets, Snowball will be fastest & cheapest (literally shipping data).

Great, thanks, here's the data plan to grab: https://aws.amazon.com/datasets/ For downloading & uploading very large datasets, would it make more sense to proxy the through Amazon to a Snowball and ship it. Very possible is the answer is it depends, just trying to get a sense of how data transfer via Amazon is works and is priced.

Also worth checking out this when thinking about speed - https://aws.amazon.com/importexport/faqs/#when-to-use

Re: Amazon S3 Transfer Acceleration

#19

Post by shortstuffsushi » Tue, Apr 19, 2016, 5:32 PM UTC

nxzero wrote: post #7

"What's a Snowball?" Animated YouTube by Amazon explains: https://youtube.com/watch?v=9uc2DSZ1wL8

Even with that, I don't get it.

"Moving lots of data either requires a huge pipe, or a ton of storage disks."

With that, they offer their Snowball device, which, if I'm understanding correctly, holds up to 50TB (now 80TB), which they physically ship to you, and then you ship back to them. How does this fix either of the constraints (disk space / connection pipe)?

Re: Amazon S3 Transfer Acceleration

#20

Post by ghshephard » Tue, Apr 19, 2016, 5:39 PM UTC

Does anybody have a sense of how fast uploading to S3 can get? I'm on a gigabit link here in Singapore, uploading to Singapore instance of S3 (via ARQ) - and I'm disappointed that I rarely see better than about 40 Mbits/second.