11–20 of 73 posts
I'd be interested to see how AWS' Elastic File System (EFS) compares (though I'd imagine it's not great, given it's mounted via NFS)
AWS has a limit on the total throughput any one account can have to S3, so the more CPUs OP adds, the worse OPs performance will be on each one. I suspect the other providers have the same restriction. I either missed it or OP didn't specify how many instances they was using at once to run their benchmark, but the more instances they used, the worse it will be per node. This did not seem to be accounted for. EDIT: OP…
This is not the case with Google Cloud Storage. I cannot speak to the other providers. Google Cloud Storage does not limit read or write throughput with the exception of our "Nearline" product (and even Nearline's limiting can be suspended for additional cost, a feature called "On-Demand I/O").
Earlier quoted context omitted.
Do you have any sources or more information about the per-account S3 limits?
I don't have any published sources, it's something they told me, but it's hinted at here: http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-... They explicitly mention the RPS per account limit in that doc, which is related.
AWS has a limit on the total throughput any one account can have to S3, so the more CPUs OP adds, the worse OPs performance will be on each one. I suspect the other providers have the same restriction. I either missed it or OP didn't specify how many instances they was using at once to run their benchmark, but the more instances they used, the worse it will be per node. This did not seem to be accounted for. EDIT: OP…
All the benchmarks were from a single instance. (Note that I have done some testing from AWS Lambda, where we had 1k lambda jobs all pulling down files from S3 at once. That's a bit harder to benchmark...)
It sounds like that wouldn't have been a factor, except for the cap you seem to have discovered on Amazon that you called out.
My only suggestion then is you may want to make it explicit that you ran the benchmarks from a single instance.
Earlier quoted context omitted.
All the benchmarks were from a single instance. (Note that I have done some testing from AWS Lambda, where we had 1k lambda jobs all pulling down files from S3 at once. That's a bit harder to benchmark...)
Hi OP, nice writeup! I hope my comment wasn't construed as dismissing the work, just a criticism of one small part. It sounds like that wouldn't have been a factor, except for the cap you seem to have discovered on Amazon that you called out. My only suggestion then is you may want to make it explicit that you ran the benchmarks from a single instance.
AWS has a limit on the total throughput any one account can have to S3, so the more CPUs OP adds, the worse OPs performance will be on each one. I suspect the other providers have the same restriction. I either missed it or OP didn't specify how many instances they was using at once to run their benchmark, but the more instances they used, the worse it will be per node. This did not seem to be accounted for. EDIT: OP…
Is this a limit that is hit anywhere near the 150GB discussed in this article, or is it something that you hit only if you are Netflix? We have TB in S3 and have not observed any limit other than EC2 instance bandwidth.
Earlier quoted context omitted.
I don't have any published sources, it's something they told me, but it's hinted at here: http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-... They explicitly mention the RPS per account limit in that doc, which is related.
RPS to S3 is limited, but not throughput to S3, except by bucket. Higher throughput can be achieved by sharding your data across multiple buckets. Also, its important to properly namespace your keys within buckets to ensure its efficiently distributed across underlying data partitions.
Earlier quoted context omitted.
RPS to S3 is limited, but not throughput to S3, except by bucket. Higher throughput can be achieved by sharding your data across multiple buckets. Also, its important to properly namespace your keys within buckets to ensure its efficiently distributed across underlying data partitions.
Unless that is a semi-recent change, that is not what I've been explicitly told. To be fair my information is at least two years old now.
Earlier quoted context omitted.
Do you have any sources or more information about the per-account S3 limits?
I don't have any published sources, it's something they told me, but it's hinted at here: http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-... They explicitly mention the RPS per account limit in that doc, which is related.