Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon S3 Transfer Acceleration

aws.amazon.com

Post reply on HN
Print view View original

41–50 of 88 posts

Re: Amazon S3 Transfer Acceleration

#41

Post by ghshephard »

philliphaydon wrote: post #39
ghshephard wrote: post #20

Does anybody have a sense of how fast uploading to S3 can get? I'm on a gigabit link here in Singapore, uploading to Singapore instance of S3 (via ARQ) - and I'm disappointed that I rarely see better than about 40 Mbits/second.

I'm in Singapore, uploading to Oregon, 100mb, took minutes , I changed to upload via signed cloud front requests, takes 15s to upload 100mb now...

I presume you mean 100 MB (or just plain 100 MBytes) - so, 53 megabits/second. So a little better, but nowhere near 1 gigabit (which I presume you have, it's so cheap here - not that it seems to be useful for anything other than great speedtest results)

Re: Amazon S3 Transfer Acceleration

#42

Post by samstave »

cmwright wrote: post #14
toomuchtodo wrote: post #2

TL;DR Newer, larger "snowball" appliance (current = 50TB and newer 80TB version), as well as access in additional international regions. Network optimized S3 uploads (faster inbound transfers) for an extra 4 cents/GB. Sidenote: I'm curious if you can get the same speedup for inbound transfers by using a Cloudfront distribution in front of your S3 bucket for no additional charge (no cost for inbound transfer on either…

We played with this a bit using both cloudfront and other cdns. While we did see some small improvements, it wasn't nearly as good as we were expecting on the upload side. It may be fixed now, but there were also many issues getting the proper headers to pass through cloudfront to s3 for chunked uploads.

This might not be the right question, but:

Did you test using S4CMD with many threads as opposed to S3CMD?

Re: Amazon S3 Transfer Acceleration

#43

Post by mabbo »

extra88 wrote: post #29
shortstuffsushi wrote: post #19

Earlier quoted context omitted.

Even with that, I don't get it. "Moving lots of data either requires a huge pipe, or a ton of storage disks." With that, they offer their Snowball device, which, if I'm understanding correctly, holds up to 50TB (now 80TB), which they physically ship to you, and then you ship back to them. How does this fix either of the constraints (disk space / connection pipe)?

With Snowball, you don't need a huge pipe because your data is being sneaker-netted to one of their data centers. You don't need a ton of storage disks because they're lending Snowball to you in the short-term as a means of copying your data to S3 (and/or Glacier) for the long-term. If you can spare a 1 gigabit connection to saturate with S3 uploads, you can send 50TB in about two weeks. It takes about a week to requ…

Does it seriously take a week to get a snowball? You'd think with all the logistics Amazon owns...

Re: Amazon S3 Transfer Acceleration

#44

Post by nxzero »

cortesoft wrote: post #32
vvanders wrote: post #23

Earlier quoted context omitted.

As they used to say, never underestimate the bandwidth of a van filled with harddrives.

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

Always IP over Avian Carriers: https://en.m.wikipedia.org/wiki/IP_over_Avian_Carriers

&&

Sneakernets

https://en.m.wikipedia.org/wiki/Sneakernet

Re: Amazon S3 Transfer Acceleration

#46

Post by twistedpair »

ghshephard wrote: post #20

Does anybody have a sense of how fast uploading to S3 can get? I'm on a gigabit link here in Singapore, uploading to Singapore instance of S3 (via ARQ) - and I'm disappointed that I rarely see better than about 40 Mbits/second.

Make sure you chunk it and use the S3 multi-part upload API. That way, you can break into N chunks, and then upload N in parallel, to get closer to saturating your uplink. The same works for downloads, where you can use byte range GET's to essentially do multi-part download.

Re: Amazon S3 Transfer Acceleration

#47

Post by wolf550e »

toomuchtodo wrote: post #36
ladon86 wrote: post #35

Earlier quoted context omitted.

I stand corrected, thanks for the info. Cloudfront doesn't appear to support multi-part uploads however, which I guess is the primary value-add of the newly announced service.

You're welcome. And you're correct. As long as you don't need to upload multi parts in parallel or objects over 5GB, standard upload works fine.

Without multipart upload, you can't resume uploads. Even with moderate sized files that's a bummer.

Re: Amazon S3 Transfer Acceleration

#50

Post by toomuchtodo »

wolf550e wrote: post #47
toomuchtodo wrote: post #36

Earlier quoted context omitted.

You're welcome. And you're correct. As long as you don't need to upload multi parts in parallel or objects over 5GB, standard upload works fine.

Without multipart upload, you can't resume uploads. Even with moderate sized files that's a bummer.

That gets less important each passing year as connections get faster, even on mobile. I agree you can't restart uploads though without multi part support.
Post reply on HN