Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

blog.zachbjornson.com

Post reply on HN
Print view View original

31–40 of 73 posts

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#31

Post by hrez »

lowbloodsugar wrote: post #23

If you are pulling large files from S3 we have found that they can be sped up by requesting multiple ranges simultaneously. It is easy to hit 5Gb/s or 10Gb/s on instances with the necessary bandwidth, accessing a single file, or multiple files. We have not encountered a limit on S3 itself. YMMV.

Excellent https://github.com/rlmcpherson/s3gof3r is my tool of choice for "fast, parallelized, pipelined streaming access to Amazon S3."

If you want to saturate network bandwidth with S3 that's the one tool I know that can do it.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#33

Post by hrez »

zbjornson wrote: post #9
jedberg wrote: post #3

AWS has a limit on the total throughput any one account can have to S3, so the more CPUs OP adds, the worse OPs performance will be on each one. I suspect the other providers have the same restriction. I either missed it or OP didn't specify how many instances they was using at once to run their benchmark, but the more instances they used, the worse it will be per node. This did not seem to be accounted for. EDIT: OP…

All the benchmarks were from a single instance. (Note that I have done some testing from AWS Lambda, where we had 1k lambda jobs all pulling down files from S3 at once. That's a bit harder to benchmark...)

Any comments on how it worked out with Lambda?

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#34

Post by hrez »

oavdeev wrote: post #24

Stock Ubuntu needs SR-IOV driver to get to the actual bandwidth limit on ec2, it makes a lot of difference. We routinely get to ~2 Gbps down from S3 with that setup (using largest instance types). edit: Gbps not GBps

Good point for "enchanced networking" instances. I didn't see OS specified in the article. AMZN linux would have SR-IOV driver by default. PV vs HVM might also have an impact.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#35

Post by rmcpherson »

oavdeev wrote: post #24

Stock Ubuntu needs SR-IOV driver to get to the actual bandwidth limit on ec2, it makes a lot of difference. We routinely get to ~2 Gbps down from S3 with that setup (using largest instance types). edit: Gbps not GBps

That's true, although the latest stock ubuntu HVM AMIs (14+, I believe) have the SR-IOV driver already and use it by default. Older AMIs need to have it installed and enabled on the AMI. I believe enhanced networking is only available on HVM amis.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#36

Post by zbjornson »

hrez wrote: post #34
oavdeev wrote: post #24

Stock Ubuntu needs SR-IOV driver to get to the actual bandwidth limit on ec2, it makes a lot of difference. We routinely get to ~2 Gbps down from S3 with that setup (using largest instance types). edit: Gbps not GBps

Good point for "enchanced networking" instances. I didn't see OS specified in the article. AMZN linux would have SR-IOV driver by default. PV vs HVM might also have an impact.

Per the comment here [1] and the linked twitter convo, I'll retest S3 with Amazon Linux soon. These tests used Ubuntu 14.04 on all providers, and did use HVM. My understanding is that this will possibly increase the network throughput of the VM, but the benchmarks stayed below the VM's capacity (which was the reason I included the charts of VM throughput).

[1] https://news.ycombinator.com/item?id=10846497

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#37

Post by rmcpherson »

In S3 tests on c3.8xlarge instances, I've seen 8 Gbps throughput on both uploads and downloads using parallelized requests. Testing with iperf between two of the same instances maxed out about 8 Gbps as well so the throughput limitation is likely EC2 networking rather than S3.

These tests were done over a year ago so bandwidth limitations on EC2 may have changed since.

This testing was with https://github.com/rlmcpherson/s3gof3r

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#38

Post by zbjornson »

hrez wrote: post #33
zbjornson wrote: post #9

Earlier quoted context omitted.

All the benchmarks were from a single instance. (Note that I have done some testing from AWS Lambda, where we had 1k lambda jobs all pulling down files from S3 at once. That's a bit harder to benchmark...)

Any comments on how it worked out with Lambda?

Reluctant to say much because the benchmarks weren't formal. However...

The throughput correlated directly with how much RAM we allocated to the Lambda function (which presumably means we were sharing the VM with fewer other jobs).

512 MB RAM, 19.5 MB/s

768 MB RAM, 29.8 MB/s

1024 MB RAM, 38.4 MB/s

1536 MB RAM, 43.7 MB/s

Note that this also used the node.js AWS SDK, which is slower to download files than some other APIs.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#39

Post by zbjornson »

rmcpherson wrote: post #37

In S3 tests on c3.8xlarge instances, I've seen 8 Gbps throughput on both uploads and downloads using parallelized requests. Testing with iperf between two of the same instances maxed out about 8 Gbps as well so the throughput limitation is likely EC2 networking rather than S3. These tests were done over a year ago so bandwidth limitations on EC2 may have changed since. This testing was with https://github.com/rlmcphe…

That's really cool. Wonder if the same technique (parallel streams) would help for Azure and GCS. I know GCS has some built-in capabilities for composite uploads/downloads, which might achieve a similar effect.
Post reply on HN