Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
blog.zachbjornson.com
Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
1–10 of 73 posts
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#2Thanks for sharing your research - I've been up to the neck in EC2 migrations and trying to benchmark as I go... S3 is the neck chunk of work. Rock on!
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#3AWS has a limit on the total throughput any one account can have to S3, so the more CPUs OP adds, the worse OPs performance will be on each one. I suspect the other providers have the same restriction.
I either missed it or OP didn't specify how many instances they was using at once to run their benchmark, but the more instances they used, the worse it will be per node.
This did not seem to be accounted for.
EDIT: OP says below it was from one instance, so what I said doesn't apply to this writeup.
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#4I'd be interested to see how AWS' Elastic File System (EFS) compares (though I'd imagine it's not great, given it's mounted via NFS)
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#5AWS has a limit on the total throughput any one account can have to S3, so the more CPUs OP adds, the worse OPs performance will be on each one. I suspect the other providers have the same restriction. I either missed it or OP didn't specify how many instances they was using at once to run their benchmark, but the more instances they used, the worse it will be per node. This did not seem to be accounted for. EDIT: OP…
Do you have any sources or more information about the per-account S3 limits?
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#6AWS has a limit on the total throughput any one account can have to S3, so the more CPUs OP adds, the worse OPs performance will be on each one. I suspect the other providers have the same restriction. I either missed it or OP didn't specify how many instances they was using at once to run their benchmark, but the more instances they used, the worse it will be per node. This did not seem to be accounted for. EDIT: OP…
Do you have any sources or more information about the per-account S3 limits?
I don't have any published sources, it's something they told me, but it's hinted at here: http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-...
They explicitly mention the RPS per account limit in that doc, which is related.
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#7AWS has a limit on the total throughput any one account can have to S3, so the more CPUs OP adds, the worse OPs performance will be on each one. I suspect the other providers have the same restriction. I either missed it or OP didn't specify how many instances they was using at once to run their benchmark, but the more instances they used, the worse it will be per node. This did not seem to be accounted for. EDIT: OP…
Do you have any sources or more information about the per-account S3 limits?
Take into account OP's former jobs. I imagine if anyone would run into such a limit, it would be Reddit or Netflix.
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#8with kernel tuning, S3 performance improves (and will probably improve on GC/Azure as well). Also, author uses Ubuntu 14.4 (see https://twitter.com/Zbjorn/status/684492084422688768), which doesn't use AWS "Enhanced networking" by default. Would be interesting to see results for tuned systems.
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#9AWS has a limit on the total throughput any one account can have to S3, so the more CPUs OP adds, the worse OPs performance will be on each one. I suspect the other providers have the same restriction. I either missed it or OP didn't specify how many instances they was using at once to run their benchmark, but the more instances they used, the worse it will be per node. This did not seem to be accounted for. EDIT: OP…
All the benchmarks were from a single instance.
(Note that I have done some testing from AWS Lambda, where we had 1k lambda jobs all pulling down files from S3 at once. That's a bit harder to benchmark...)
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#10AWS has a limit on the total throughput any one account can have to S3, so the more CPUs OP adds, the worse OPs performance will be on each one. I suspect the other providers have the same restriction. I either missed it or OP didn't specify how many instances they was using at once to run their benchmark, but the more instances they used, the worse it will be per node. This did not seem to be accounted for. EDIT: OP…
This is not the case with Google Cloud Storage. I cannot speak to the other providers.
Google Cloud Storage does not limit read or write throughput with the exception of our "Nearline" product (and even Nearline's limiting can be suspended for additional cost, a feature called "On-Demand I/O").