Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage • Hacker News Board

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#71

Post by acdha » Wed, Jan 06, 2016, 3:27 PM UTC

zbjornson wrote: post #11

dwelch2344 wrote: post #4

I'd be interested to see how AWS' Elastic File System (EFS) compares (though I'd imagine it's not great, given it's mounted via NFS)

I've been on the list to get into their preview program for a while so I can benchmark it, actually! Part 3 of the blog post is going to include some NFS stuff either way.

When you do, it would be really useful to include the classic fio/bonnie/etc. stuff to break down performance by the type of operation (e.g. file creation / deletion, streaming read/write, random read/write) and block size.

EFS supports NFSv4 so it should avoid being as routinely limited by server round-trip latency as NFSv3 tends to be but it'd be nice to see how well that works in practice.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#72

Post by zbjornson » Wed, Jan 06, 2016, 7:51 PM UTC

mistermann wrote: post #68

skywhopper wrote: post #67

Very interesting comparison, glad to see it. I don't have a comment on the content itself but I do have a note on the presentation. The colors used for S3 and Azure Storage in the graphs are very near indistiguishable to me, as I have moderate red-green colorblindness. It's easier to tell apart on the bar graphs, since the patches of color are much larger, although I still have to work at it, and use the hints of the…

Color blind here as well, I had to zoom in incredibly close to distinguish the difference.

Thanks for pointing this out, and my apologies! Will fix that going forward.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#73

Post by zbjornson » Wed, Jan 06, 2016, 8:07 PM UTC

qaq wrote: post #65

semi-extrinsic wrote: post #64

Earlier quoted context omitted.

The big question is, why does it need to run in 10s? The main reason I can see is to be able to run this analysis very frequently, but then your workload is approaching constant. The total amount of data is 150 GB; that would easily fit into memory on a single powerful 2-socket server with 20 cores and would then run in less than 15 minutes. The hardware required to do that will cost you ~ $6000 from Dell; assuming a…

Vcore is hyperthread of unknown CPU. So in reality 1000 vcores is 500 real cores. - All the overheads it's more like 450 given the low utilization until dataset loads to keep it at 10 sec you would need 90 real cores or 4 X 3 node dual boxes (ebay 1.5K each) and 2 X infiniband switches (ebay 2X300). For 6600 you have a dedicated solution with no latency bubbles fixed low cost.

Briefly... We have many data sets, and the We've considered a few kinds of platforms (AWS spot fleet/GCE autoscaled preemptible VMs, AWS Lambda, bare metal hosting, even Beowulf clusters), and while bare metal has its benefits as you've pointed out, at our current stage it doesn't make sense for us financially.

I omitted from the blog post that we don't rely exclusively on object storage services because its performance is relatively low. We cache files on compute nodes so we avoid that "80% of time is spent reading data" a lot of the time.

(Re: Netflix, in qaq's other comment, I don't have a hard number for this, but I thought a typical AWS data center is only under 20-30% load at any given time.)