Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

blog.zachbjornson.com

Post reply on HN
Print view View original

61–70 of 73 posts

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#61

Post by cottonseed »

qaq wrote: post #58
cottonseed wrote: post #57

Earlier quoted context omitted.

Because renting 1000 cores for a limited time is much cheaper than buying them outright?

1000 cores of what ? Vcore is marketing BS. Even if it was not marketing BS it's 28 2U 3 node boxes (if using older cpus) or 14 2U 3 node boxes (if using more recent ones) unless they have extremely spiky workload using AWS is pointless. Bandwidth bound scientific apps ==> use infiniband cluster.

The OP is talking about running $0.027 worth of computation (1000 cores for 10s at 0.01/core/hr) and you think he should spend tens of thousands on hardware?

I'm not doubting a custom build will give him much greater bandwidth. I just doubt the workload has to be "extremely" spiky to make the cloud cost-effective.

Of course, he's going to get billed for 10m or 1hr minimum (Google or Amazon), so that's assuming he can amortize his startup across multiple jobs.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#63

Post by qaq »

cottonseed wrote: post #61
qaq wrote: post #58

Earlier quoted context omitted.

1000 cores of what ? Vcore is marketing BS. Even if it was not marketing BS it's 28 2U 3 node boxes (if using older cpus) or 14 2U 3 node boxes (if using more recent ones) unless they have extremely spiky workload using AWS is pointless. Bandwidth bound scientific apps ==> use infiniband cluster.

The OP is talking about running $0.027 worth of computation (1000 cores for 10s at 0.01/core/hr) and you think he should spend tens of thousands on hardware? I'm not doubting a custom build will give him much greater bandwidth. I just doubt the workload has to be "extremely" spiky to make the cloud cost-effective. Of course, he's going to get billed for 10m or 1hr minimum (Google or Amazon), so that's assuming he can…

They have a single client running a single 10 sec job in a day? They plan to continue having a single client running a single 10 sec job in a day? The workload does have to be spiky to make the cloud cost-effective. There are workloads which are not appropriate for AWS. For any serious client AWS is a bad idea simply because there is single tenant (Netflix) consuming such a high percentage of resource that if they make a mistake causing a 40-50% increase in their load everyone gets f#$%ed.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#64

Post by semi-extrinsic »

cottonseed wrote: post #61
qaq wrote: post #58

Earlier quoted context omitted.

1000 cores of what ? Vcore is marketing BS. Even if it was not marketing BS it's 28 2U 3 node boxes (if using older cpus) or 14 2U 3 node boxes (if using more recent ones) unless they have extremely spiky workload using AWS is pointless. Bandwidth bound scientific apps ==> use infiniband cluster.

The OP is talking about running $0.027 worth of computation (1000 cores for 10s at 0.01/core/hr) and you think he should spend tens of thousands on hardware? I'm not doubting a custom build will give him much greater bandwidth. I just doubt the workload has to be "extremely" spiky to make the cloud cost-effective. Of course, he's going to get billed for 10m or 1hr minimum (Google or Amazon), so that's assuming he can…

The big question is, why does it need to run in 10s? The main reason I can see is to be able to run this analysis very frequently, but then your workload is approaching constant.

The total amount of data is 150 GB; that would easily fit into memory on a single powerful 2-socket server with 20 cores and would then run in less than 15 minutes. The hardware required to do that will cost you ~ $6000 from Dell; assuming a system lifetime of five years and assuming (like you do) that you can amortize across multiple jobs, the cost is roughly the same as from the cloud, about $0.036 per analysis.

I'm fairly certain that, in the end, it's not more expensive for the customer to just buy a server to run the analysis on.

Edit: I see OP says 80% of the time is spent reading data into memory, at about 100 MB/s. Add $500 worth of SSD to the example server I outlined, and we can cut the application runtime by >70%, making the dedicated hardware significantly cheaper.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#65

Post by qaq »

semi-extrinsic wrote: post #64
cottonseed wrote: post #61

Earlier quoted context omitted.

The OP is talking about running $0.027 worth of computation (1000 cores for 10s at 0.01/core/hr) and you think he should spend tens of thousands on hardware? I'm not doubting a custom build will give him much greater bandwidth. I just doubt the workload has to be "extremely" spiky to make the cloud cost-effective. Of course, he's going to get billed for 10m or 1hr minimum (Google or Amazon), so that's assuming he can…

The big question is, why does it need to run in 10s? The main reason I can see is to be able to run this analysis very frequently, but then your workload is approaching constant. The total amount of data is 150 GB; that would easily fit into memory on a single powerful 2-socket server with 20 cores and would then run in less than 15 minutes. The hardware required to do that will cost you ~ $6000 from Dell; assuming a…

Vcore is hyperthread of unknown CPU. So in reality 1000 vcores is 500 real cores. - All the overheads it's more like 450 given the low utilization until dataset loads to keep it at 10 sec you would need 90 real cores or 4 X 3 node dual boxes (ebay 1.5K each) and 2 X infiniband switches (ebay 2X300). For 6600 you have a dedicated solution with no latency bubbles fixed low cost.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#66

Post by slagfart »

qaq wrote: post #63
cottonseed wrote: post #61

Earlier quoted context omitted.

The OP is talking about running $0.027 worth of computation (1000 cores for 10s at 0.01/core/hr) and you think he should spend tens of thousands on hardware? I'm not doubting a custom build will give him much greater bandwidth. I just doubt the workload has to be "extremely" spiky to make the cloud cost-effective. Of course, he's going to get billed for 10m or 1hr minimum (Google or Amazon), so that's assuming he can…

They have a single client running a single 10 sec job in a day? They plan to continue having a single client running a single 10 sec job in a day? The workload does have to be spiky to make the cloud cost-effective. There are workloads which are not appropriate for AWS. For any serious client AWS is a bad idea simply because there is single tenant (Netflix) consuming such a high percentage of resource that if they ma…

You're hypothesising about something that has never happened. Check out some 3rd party cloud uptime metrics - the major providers (AWS, Google, Azure) have had less than an hour of downtime in the past year. Reliability is no longer on the agenda - it has been proven.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#67

Post by skywhopper »

Very interesting comparison, glad to see it. I don't have a comment on the content itself but I do have a note on the presentation.

The colors used for S3 and Azure Storage in the graphs are very near indistiguishable to me, as I have moderate red-green colorblindness. It's easier to tell apart on the bar graphs, since the patches of color are much larger, although I still have to work at it, and use the hints of the labels, but on the line graphs, it's basically impossible to tell apart. A darker shade of green would solve the problem for me personally, but I'm not all that bad a case, nor an expert on the best shades to pick for general color-blindness accessibility.

Just something to think about when presenting data like this.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#68

Post by mistermann »

skywhopper wrote: post #67

Very interesting comparison, glad to see it. I don't have a comment on the content itself but I do have a note on the presentation. The colors used for S3 and Azure Storage in the graphs are very near indistiguishable to me, as I have moderate red-green colorblindness. It's easier to tell apart on the bar graphs, since the patches of color are much larger, although I still have to work at it, and use the hints of the…

Color blind here as well, I had to zoom in incredibly close to distinguish the difference.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#69

Post by zeristor »

dharbin wrote: post #50
jen20 wrote: post #40

Has the author (if they are reading here) considered using Joyent's Manta to take the processing to the data instead?

I don't know about Manta, but this is the entire point of HDFS. It easier to move code than data.

Indeed, but they're having such fun. Let's leave them be.

Re: Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage

#70

Post by qaq »

slagfart wrote: post #66
qaq wrote: post #63

Earlier quoted context omitted.

They have a single client running a single 10 sec job in a day? They plan to continue having a single client running a single 10 sec job in a day? The workload does have to be spiky to make the cloud cost-effective. There are workloads which are not appropriate for AWS. For any serious client AWS is a bad idea simply because there is single tenant (Netflix) consuming such a high percentage of resource that if they ma…

You're hypothesising about something that has never happened. Check out some 3rd party cloud uptime metrics - the major providers (AWS, Google, Azure) have had less than an hour of downtime in the past year. Reliability is no longer on the agenda - it has been proven.

It did happen and my clients were affected. After AWS fu$%up rollout of software update in 2011 that overvelmed their control plane and had whole zones down for many hours and took many days to fully restore, they rolled out patches that throttle cross zone migration. After those patches at one point Netflix was having issues and started massive migration that hit throttle thresholds and affected ability of other tenants to move to non affected zones. It's very far from hypothetical given netflix consumes about 30% of resources (which translates to many whole size physical datacenters) if they spike 50% they will overvelm the spare capacity.
Post reply on HN