Spark Breaks Previous Large-Scale Sort Record
databricks.com
Spark Breaks Previous Large-Scale Sort Record
1–10 of 58 posts
Re: Spark Breaks Previous Large-Scale Sort Record
#2Most recent results I can see to compare to (Google, Yahoo, Quantcast):
https://www.quantcast.com/inside-quantcast/2013/12/petabyte-...
Re: Spark Breaks Previous Large-Scale Sort Record
#3For the curious, the (max) price of those instances is $6.82/hr, so 206 * 6.82 * (23/60) = $538.55 --If they did it with non-reserved instances in US East.
If they used reserved instances in USEast, it drops to $181.
Obviously there are lots of costs involved beside the final perfect run, but it's an interesting ballpark.
Re: Spark Breaks Previous Large-Scale Sort Record
#4For the curious, the (max) price of those instances is $6.82/hr, so 206 * 6.82 * (23/60) = $538.55 --If they did it with non-reserved instances in US East. If they used reserved instances in USEast, it drops to $181. Obviously there are lots of costs involved beside the final perfect run, but it's an interesting ballpark.
You have to put spaces around your * 's to keep HN from italicizing everything.
Re: Spark Breaks Previous Large-Scale Sort Record
#5It's interesting, but not earth-shattering. The "10x fewer nodes" means nothing; how powerful are the new nodes? What's the network? Do you use SSDs? etc. etc.
They also tuned their code to this specific problem:
"Exploiting Cache Locality: In the sort benchmark, each record is 100 bytes, where the sort key is the first 10 bytes. As we were profiling our sort program, we noticed the cache miss rate was high, because each comparison required an object pointer lookup that was random..... Combining TimSort with our new layout to exploit cache locality, the CPU time for sorting was reduced by a factor of 5."
I would love to see MR and Spark compete on the exact same hardware configuration.
Re: Spark Breaks Previous Large-Scale Sort Record
#6For the curious, the (max) price of those instances is $6.82/hr, so 206 * 6.82 * (23/60) = $538.55 --If they did it with non-reserved instances in US East. If they used reserved instances in USEast, it drops to $181. Obviously there are lots of costs involved beside the final perfect run, but it's an interesting ballpark.
You have to put spaces around your * 's to keep HN from italicizing everything.
Oops, edited. Thanks!
Re: Spark Breaks Previous Large-Scale Sort Record
#7It's interesting, but not earth-shattering. The "10x fewer nodes" means nothing; how powerful are the new nodes? What's the network? Do you use SSDs? etc. etc. They also tuned their code to this specific problem: " Exploiting Cache Locality: In the sort benchmark, each record is 100 bytes, where the sort key is the first 10 bytes. As we were profiling our sort program, we noticed the cache miss rate was high, because…
The article says exactly what they ran on. EC2 i2.8xlarge instances which have 32 cores, 800GB SSD and 244GB RAM.
Re: Spark Breaks Previous Large-Scale Sort Record
#8It's interesting, but not earth-shattering. The "10x fewer nodes" means nothing; how powerful are the new nodes? What's the network? Do you use SSDs? etc. etc. They also tuned their code to this specific problem: " Exploiting Cache Locality: In the sort benchmark, each record is 100 bytes, where the sort key is the first 10 bytes. As we were profiling our sort program, we noticed the cache miss rate was high, because…
The article says exactly what they ran on. EC2 i2.8xlarge instances which have 32 cores, 800GB SSD and 244GB RAM.
I read that. But how does that compare with the nodes they're comparing against ("10x fewer nodes")?
Re: Spark Breaks Previous Large-Scale Sort Record
#9It's interesting, but not earth-shattering. The "10x fewer nodes" means nothing; how powerful are the new nodes? What's the network? Do you use SSDs? etc. etc. They also tuned their code to this specific problem: " Exploiting Cache Locality: In the sort benchmark, each record is 100 bytes, where the sort key is the first 10 bytes. As we were profiling our sort program, we noticed the cache miss rate was high, because…
The article says exactly what they ran on. EC2 i2.8xlarge instances which have 32 cores, 800GB SSD and 244GB RAM.
Also used the "Enhanced Networking" option on the instances which means single root I/O virtualization underneath.
Re: Spark Breaks Previous Large-Scale Sort Record
#10The 100 terabyte benchmark used 206 Spark nodes, compared with 2100 Hadoop nodes.
Going up to 1 petabyte, the Hadoop comparison adds more nodes, 3800, while the Spark benchmark actually reduced the number of nodes to 190.
Does Spark scale well beyond ~200 nodes, or does the network become the bottleneck?
In any case, it's an impressive result considering that they didn't use Spark's in-memory cache.