When I see things like "data set size 150GB" and "1000 CPUS" I just naturally assume they are all in memory and never come from disk :-)
I think that data set is too small to constitute a good benchmark for the setup.
Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
21–30 of 73 posts
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#22Earlier quoted context omitted.
I don't have any published sources, it's something they told me, but it's hinted at here: http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-... They explicitly mention the RPS per account limit in that doc, which is related.
Can you quote the piece where they mention RPS per account limit because I cannot find it.
> However, if you expect a rapid increase in the request rate for a bucket to more than 300 PUT/LIST/DELETE requests per second or more than 800 GET requests per second, we recommend that you open a support case to prepare for the workload and avoid any temporary limits on your request rate.
You have to know how to read their docs. :) This is basically code for, "there is a default limit here that you have to get raised if you want to go above it".
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#23If you are pulling large files from S3 we have found that they can be sped up by requesting multiple ranges simultaneously. It is easy to hit 5Gb/s or 10Gb/s on instances with the necessary bandwidth, accessing a single file, or multiple files. We have not encountered a limit on S3 itself. YMMV.
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#24Stock Ubuntu needs SR-IOV driver to get to the actual bandwidth limit on ec2, it makes a lot of difference. We routinely get to ~2 Gbps down from S3 with that setup (using largest instance types).
edit: Gbps not GBps
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#25I'd be interested to see how AWS' Elastic File System (EFS) compares (though I'd imagine it's not great, given it's mounted via NFS)
No hard numbers for you, but FWIW I ran tests about 4 months ago and the performance was /very/ low compared to what is achievable compared to S3 and even normal NAS.
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#26AWS has a limit on the total throughput any one account can have to S3, so the more CPUs OP adds, the worse OPs performance will be on each one. I suspect the other providers have the same restriction. I either missed it or OP didn't specify how many instances they was using at once to run their benchmark, but the more instances they used, the worse it will be per node. This did not seem to be accounted for. EDIT: OP…
If such a limit exists, it would not have been hit on such a small benchmark. However, I am unaware of any such limit and it has never been raised in any discussion I have had with them. I am responsible for a large compute and data storage platform backed by S3. Is this a limit that is hit anywhere near the 150GB discussed in this article, or is it something that you hit only if you are Netflix? We have TB in S3 and…
The amount of data one has in S3 isn't really relevant to the discussion, only how quickly you're trying to pull it into your instances.
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#27When I see things like "data set size 150GB" and "1000 CPUS" I just naturally assume they are all in memory and never come from disk :-)
That's one of many data sets on the server, so unfortunately we can't keep them all in memory at once. :(
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#28When I see things like "data set size 150GB" and "1000 CPUS" I just naturally assume they are all in memory and never come from disk :-)
I think that data set is too small to constitute a good benchmark for the setup.
You're not wrong, but apparently such a short burst is what they're actually doing in their application.
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#29Earlier quoted context omitted.
Can you quote the piece where they mention RPS per account limit because I cannot find it.
> However, if you expect a rapid increase in the request rate for a bucket to more than 300 PUT/LIST/DELETE requests per second or more than 800 GET requests per second, we recommend that you open a support case to prepare for the workload and avoid any temporary limits on your request rate. You have to know how to read their docs. :) This is basically code for, "there is a default limit here that you have to get rai…
The full quote is:
>Amazon S3 scales to support very high request rates. If your request rate grows steadily, Amazon S3 automatically partitions your buckets as needed to support higher request rates. However, if you expect a rapid increase in the request rate for a bucket to more than 300 PUT/LIST/DELETE requests per second or more than 800 GET requests per second, we recommend that you open a support case to prepare for the workload and avoid any temporary limits on your request rate. To open a support case, go to Contact Us.
So this looks like an auto scaling issue. It states "S3 automatically scales to support higher request rates". However, if we know that a bucket is going to need to scale dramatically, we can request, in advance, that the S3 team pre-scales it.
I'm sure there is an account limit, but to run 1000 cpu's already requires requesting an increase in the account's EC2 instance limit. Are you saying that a team trying to access 150Gb of files, or to make 1000 RPS, as the article documents, will hit that limit? From your experience, how big is this hard limit? Is it Netflix scale or is it GB or TB?
Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
#30Earlier quoted context omitted.
If such a limit exists, it would not have been hit on such a small benchmark. However, I am unaware of any such limit and it has never been raised in any discussion I have had with them. I am responsible for a large compute and data storage platform backed by S3. Is this a limit that is hit anywhere near the 150GB discussed in this article, or is it something that you hit only if you are Netflix? We have TB in S3 and…
The amount of data one has in S3 isn't really relevant to the discussion, only how quickly you're trying to pull it into your instances.
Ok then let me rephrase: Is this a limit that is hit anywhere near the 603GB/s figure in this article, or is it something that you hit only if you are Netflix? You seem to be claiming that such a limit exists and that you know what it is. Can you share or is this NDA territory?