11–20 of 58 posts
For the curious, the (max) price of those instances is $6.82/hr, so 206 * 6.82 * (23/60) = $538.55 --If they did it with non-reserved instances in US East. If they used reserved instances in USEast, it drops to $181. Obviously there are lots of costs involved beside the final perfect run, but it's an interesting ballpark.
http://googleresearch.blogspot.com/2011/09/sorting-petabytes...
Earlier quoted context omitted.
The article says exactly what they ran on. EC2 i2.8xlarge instances which have 32 cores, 800GB SSD and 244GB RAM.
I read that. But how does that compare with the nodes they're comparing against ("10x fewer nodes")?
The network part is probably the most important one here, and both have comparable network.
The 100 terabyte benchmark used 206 Spark nodes, compared with 2100 Hadoop nodes. Going up to 1 petabyte, the Hadoop comparison adds more nodes, 3800, while the Spark benchmark actually reduced the number of nodes to 190. Does Spark scale well beyond ~200 nodes, or does the network become the bottleneck? In any case, it's an impressive result considering that they didn't use Spark's in-memory cache.
Spark runs fine on thousands of nodes.
The 100 terabyte benchmark used 206 Spark nodes, compared with 2100 Hadoop nodes. Going up to 1 petabyte, the Hadoop comparison adds more nodes, 3800, while the Spark benchmark actually reduced the number of nodes to 190. Does Spark scale well beyond ~200 nodes, or does the network become the bottleneck? In any case, it's an impressive result considering that they didn't use Spark's in-memory cache.
> [O]ur Spark cluster was able to sustain ... 1.1 GB/s/node network activity during the reduce phase, saturating the 10Gbps link available on these machines.
If the network is the bottleneck it makes sense to reduce the number of nodes to reduce the network communications.
The 100 terabyte benchmark used 206 Spark nodes, compared with 2100 Hadoop nodes. Going up to 1 petabyte, the Hadoop comparison adds more nodes, 3800, while the Spark benchmark actually reduced the number of nodes to 190. Does Spark scale well beyond ~200 nodes, or does the network become the bottleneck? In any case, it's an impressive result considering that they didn't use Spark's in-memory cache.
I believe the network had become a bottleneck. As per the article: > [O]ur Spark cluster was able to sustain ... 1.1 GB/s/node network activity during the reduce phase, saturating the 10Gbps link available on these machines. If the network is the bottleneck it makes sense to reduce the number of nodes to reduce the network communications.