Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Spark Breaks Previous Large-Scale Sort Record

databricks.com

Post reply on HN
Print view View original

31–40 of 58 posts

Re: Spark Breaks Previous Large-Scale Sort Record

#31

Post by gtrubetskoy »

metronius wrote: post #26
gtrubetskoy wrote: post #25

Earlier quoted context omitted.

Hadoop != M/R, FWIW. M/R support is left in Yarn for backwards compatibility mostly. If by M/R you mean Hadoop - Cloudera has done no such thing, their largest customer base is Hadoop. As to "paradigm shift", we're so early in this that I don't think there even is a paradigm to shift.

I mean M/R by M/R ;) http://vision.cloudera.com/mapreduce-spark/

Sure, "we're 100% behind Impala", "oops, sorry, now it's Spark" - give them a few months and they'll change their mind to something else again. :)

Re: Spark Breaks Previous Large-Scale Sort Record

#32

Post by deeviant »

chubot wrote: post #13

FWIW, in 2011, Google wrote that they achieved a PB sort in 33 minutes on 8000 computers, vs. 234 minutes on 190 computers with 6080 cores reported by Spark here. http://googleresearch.blogspot.com/2011/09/sorting-petabytes...

I'm not sure why you list Google as using "8000 computers" and Spark using "190 computers with 6080 cores".

Using two different metrics for two like things seems like there is some sort of implication there. Were Google's machines single-cored?

Re: Spark Breaks Previous Large-Scale Sort Record

#33

Post by discardorama »

rxin wrote: post #14
discardorama wrote: post #8

Earlier quoted context omitted.

I read that. But how does that compare with the nodes they're comparing against ("10x fewer nodes")?

The old entry had 10Gb/s (40 nodes/rack 160Gbps rack to spine. 2.5:1 subscription), 64GB of RAM, and 12 x 3TB SATA. The network part is probably the most important one here, and both have comparable network.

Since each node was handling 500GB of data (roughly), I think the disk speed may have been a more critical factor since each node had 244GB of memory. Their nodes used SSDs; the older nodes used spinning rust. The seek times alone will be a killer.

Re: Spark Breaks Previous Large-Scale Sort Record

#34

Post by discardorama »

saryant wrote: post #7
discardorama wrote: post #5

It's interesting, but not earth-shattering. The "10x fewer nodes" means nothing; how powerful are the new nodes? What's the network? Do you use SSDs? etc. etc. They also tuned their code to this specific problem: " Exploiting Cache Locality: In the sort benchmark, each record is 100 bytes, where the sort key is the first 10 bytes. As we were profiling our sort program, we noticed the cache miss rate was high, because…

The article says exactly what they ran on. EC2 i2.8xlarge instances which have 32 cores, 800GB SSD and 244GB RAM.

Not just 800GB of SSD; 8x 800GB of SSD!

Re: Spark Breaks Previous Large-Scale Sort Record

#35

Post by rxin »

discardorama wrote: post #33
rxin wrote: post #14

Earlier quoted context omitted.

The old entry had 10Gb/s (40 nodes/rack 160Gbps rack to spine. 2.5:1 subscription), 64GB of RAM, and 12 x 3TB SATA. The network part is probably the most important one here, and both have comparable network.

Since each node was handling 500GB of data (roughly), I think the disk speed may have been a more critical factor since each node had 244GB of memory. Their nodes used SSDs; the older nodes used spinning rust. The seek times alone will be a killer.

Not sure why you mentioned seek time. In large scale, distributed sorting, I/O is mostly sequential.

Re: Spark Breaks Previous Large-Scale Sort Record

#37

Post by trhway »

discardorama wrote: post #34
saryant wrote: post #7

Earlier quoted context omitted.

The article says exactly what they ran on. EC2 i2.8xlarge instances which have 32 cores, 800GB SSD and 244GB RAM.

Not just 800GB of SSD; 8x 800GB of SSD!

yes, that's the key. The local IO on previous reported systems was HDDs, thus 300-600Mb/s at best with 6 drives per machine, while this guys are getting 3.3Gb/s. Getting full performance - 1.1Gb/s - of 10Gb network - were they lucky or AWS now is that good ? (couple years ago i was able to get only 400Mb/s node-to-node on cluster compute nodes there)

Re: Spark Breaks Previous Large-Scale Sort Record

#38

Post by chubot »

deeviant wrote: post #32
chubot wrote: post #13

FWIW, in 2011, Google wrote that they achieved a PB sort in 33 minutes on 8000 computers, vs. 234 minutes on 190 computers with 6080 cores reported by Spark here. http://googleresearch.blogspot.com/2011/09/sorting-petabytes...

I'm not sure why you list Google as using "8000 computers" and Spark using "190 computers with 6080 cores". Using two different metrics for two like things seems like there is some sort of implication there. Were Google's machines single-cored?

I'm just writing down exactly what they reported. They used different metrics.

Certainly it would be interesting to have an apples to apples comparison. But the computers aren't the only thing that is relevant -- we also need to know about the networking hardware.

Re: Spark Breaks Previous Large-Scale Sort Record

#39

Post by rxin »

ch wrote: post #36
rxin wrote: post #16

Thanks for sharing this. I'm the author of this blog post. Free free to ask me anything.

Curious if using the Sparrow scheduler would have been a net gain/loss to this type of work load?

It would help a little bit (maybe a few percent), but not much because the scheduling latency was relatively low for these tasks (the largest scheduling delay was ~10 secs, whereas each task takes minutes).
Post reply on HN