Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

blog.zachbjornson.com

Post reply on HN
Print view View original

51–60 of 73 posts

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#51

Post by zbjornson »

hrez wrote: post #48
zbjornson wrote: post #41

Earlier quoted context omitted.

Network was VPC. The EC2 instance had an IG attached, yes, but I'm not sure if you're asking if an internal vs. external URL for S3 was used? Are you saying there's a better endpoint than s3- .amazonaws.com for S3 requests from EC2?

I meant http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-en... It's a private connection to AWS services including S3. You'd use the same URL as it's a routing basically. No idea if VPC endpoints would be better than IG though. P.S. Just tested and I get about half of the latency on VPC endpoint.

Neat, did not know about that. Will add it to my follow-up benchmarks. Thanks for all the comments. :)

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#52

Post by vtuulos »

lowbloodsugar wrote: post #29
jedberg wrote: post #22

Earlier quoted context omitted.

> However, if you expect a rapid increase in the request rate for a bucket to more than 300 PUT/LIST/DELETE requests per second or more than 800 GET requests per second, we recommend that you open a support case to prepare for the workload and avoid any temporary limits on your request rate. You have to know how to read their docs. :) This is basically code for, "there is a default limit here that you have to get rai…

The full quote is: >Amazon S3 scales to support very high request rates. If your request rate grows steadily, Amazon S3 automatically partitions your buckets as needed to support higher request rates. However, if you expect a rapid increase in the request rate for a bucket to more than 300 PUT/LIST/DELETE requests per second or more than 800 GET requests per second, we recommend that you open a support case to prepar…

We are routinely pulling a dataset of hundreds of GBs to 100+ instances (1600+ cores) in parallel. We have never noticed throughput going down with the number of nodes. S3 delivers the maximum throughput of 2-4Gbps / instance very consistently.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#53

Post by oavdeev »

rmcpherson wrote: post #35
oavdeev wrote: post #24

Stock Ubuntu needs SR-IOV driver to get to the actual bandwidth limit on ec2, it makes a lot of difference. We routinely get to ~2 Gbps down from S3 with that setup (using largest instance types). edit: Gbps not GBps

That's true, although the latest stock ubuntu HVM AMIs (14+, I believe) have the SR-IOV driver already and use it by default. Older AMIs need to have it installed and enabled on the AMI. I believe enhanced networking is only available on HVM amis.

This problem definitely existed with official 14.04 (HVM) AMI, though I haven't re-tested this recently, they may have fixed it. It did have some kind of SR-IOV driver but it was too old.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#54

Post by hiyer »

zbjornson wrote: post #36
hrez wrote: post #34

Earlier quoted context omitted.

Good point for "enchanced networking" instances. I didn't see OS specified in the article. AMZN linux would have SR-IOV driver by default. PV vs HVM might also have an impact.

Per the comment here [1] and the linked twitter convo, I'll retest S3 with Amazon Linux soon. These tests used Ubuntu 14.04 on all providers, and did use HVM. My understanding is that this will possibly increase the network throughput of the VM, but the benchmarks stayed below the VM's capacity (which was the reason I included the charts of VM throughput). [1] https://news.ycombinator.com/item?id=10846497

Couple of other points:

1. Enhanced Networking (SRIOV) only works in a VPC and not in EC2-Classic.

2. I think the 4x instances don't support 10Gb ethernet. If that is the case, it would also be instructive to test the 8x instances on S3.

For some very application (Hadoop) specific tests of Enhanced Networking, please take a look at https://www.qubole.com/blog/product/hadoop-enhanced-networki...

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#55

Post by ChuckMcM »

zbjornson wrote: post #27
ChuckMcM wrote: post #17

When I see things like "data set size 150GB" and "1000 CPUS" I just naturally assume they are all in memory and never come from disk :-)

That's one of many data sets on the server, so unfortunately we can't keep them all in memory at once. :(

Lets assume when you're saying "cpu" when you mean "core" and your typical server class machine has 24 of those. A 1000 "cpus" is 41 machines, if they each donate 32GB to the cause[1] that is 1.3TB worth of data which is only a few microseconds away from any core.

I'm not sure why anyone would build a server with less than 96GB on it these days, so its not at all unreasonable. Now your service provider my jerk you around but you can run two racks of machines (48 machines) in a data center with specs like that for about $25K/month (including dual gigabit network pipes to your favorite IP transit provider) So it isn't even all that huge of an investment.

[1] Consider your typical 'memcached' type service where data is named as a function of IP and offset.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#58

Post by qaq »

cottonseed wrote: post #57
qaq wrote: post #56

WTF would one deploy such thing in the cloud?

Because renting 1000 cores for a limited time is much cheaper than buying them outright?

1000 cores of what ? Vcore is marketing BS. Even if it was not marketing BS it's 28 2U 3 node boxes (if using older cpus) or 14 2U 3 node boxes (if using more recent ones) unless they have extremely spiky workload using AWS is pointless. Bandwidth bound scientific apps ==> use infiniband cluster.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#59

Post by qaq »

zbjornson wrote: post #46
vgt wrote: post #43

Earlier quoted context omitted.

There are plenty of architectures that do exactly this. EMR-on-S3, Google Dataproc on GCS, Snowflake-on-S3, BigQuery-on-GCS, etc etc. The bigger point in the article is that these exact "take processing to the data" architectures operate exceedingly well on S3, GCS, Azure. And, as a biased observer, these architectures operate on GCS the best due to great performance measured in the article, quick VM standup times, l…

I'm still trying to parse the docs and Manta source code to see what it actually does, but it seems unique if the data storage nodes are also the data processing nodes and no data transfer happens from some storage service before the job begins. The other key factor is having neither startup time nor the cost of a perpetually running cluster. Per my comment below [1], we have used Lambda with S3 to get something like…

Not only that but the thing is built by guys who really know what they are doing like Bryan Cantrill and other former SUN top people.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#60

Post by gtaylor »

qaq wrote: post #56

WTF would one deploy such thing in the cloud?

I spun up something like 200 "cores" to archive a large Cassandra cluster to Google Storage (Kubernetes cluster plus 200+ containers running the archive worker). Could have gone much bigger to get it done faster, but it wasn't necessary. ETL or archive jobs would be the most common case, to answer your question.
Post reply on HN