Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data sorting world record: 1 terabyte, 1 minute

scienceblog.com

Post reply on HN
Print view View original

1–10 of 16 posts

Re: Data sorting world record: 1 terabyte, 1 minute

#4

Post by nl »

That's pretty impressive. Back in 2008, Google did a 1TB sort in 68 seconds (http://googleblog.blogspot.com/2008/11/sorting-1pb-with-mapr...), breaking the previous record of 209 seconds (which I think was done on Yahoo's Hadoop cluster)

I expect these aren't quite comparable (the op link talks about it being non-generalized data), but it's interesting how these high performance numbers are becoming more achievable.

Re: Data sorting world record: 1 terabyte, 1 minute

#6

Post by dasil003 »

stuff4ben wrote: post #5

it's hard to infer if the record is being set due to better hardware or better algorithms. I'd like to see benchmark improvements in algorithms not hardware, but I'm a software guy.

They imply that it's largely software, but obviously the hardware is not the same. If they actually reduced power consumption by 75% then it's pretty interesting regardless of what the mix is.

Re: Data sorting world record: 1 terabyte, 1 minute

#8

Post by dadkins »

stuff4ben wrote: post #5

it's hard to infer if the record is being set due to better hardware or better algorithms. I'd like to see benchmark improvements in algorithms not hardware, but I'm a software guy.

It's hardware. They have have 52 servers, each with 16 disks which I assume are capable of I/O at 100MB/s. That's an aggregate disk I/O rate of 83.2 GB/s. Their network switch provides 10Gbps connectivity between each pair, which implies they can move data between nodes at 65 GB/s. With that hardware setup they can read 1TB from disk and distribute in around 15 seconds. They can write sorted data back to disk in 12 seconds. That leaves a little over 30 seconds for each node to sort 19.2 GB in memory. But each node has 8 cores, so each core has to sort 2.4 GB in 30 seconds. I've seen Burstsort chew through that amount of data in half the time.

Re: Data sorting world record: 1 terabyte, 1 minute

#9

Post by javanix »

stuff4ben wrote: post #5

it's hard to infer if the record is being set due to better hardware or better algorithms. I'd like to see benchmark improvements in algorithms not hardware, but I'm a software guy.

I was under the impression (could be remembering something wrong, however) that sorting algorithms had a provable (or at least strongly believed) best-case of O(nlogn), and that we already have algorithms that meet that.

If that is correct, most improvements would probably come from hardware and software use-case tuning.

Re: Data sorting world record: 1 terabyte, 1 minute

#10

Post by sp332 »

javanix wrote: post #9
stuff4ben wrote: post #5

it's hard to infer if the record is being set due to better hardware or better algorithms. I'd like to see benchmark improvements in algorithms not hardware, but I'm a software guy.

I was under the impression (could be remembering something wrong, however) that sorting algorithms had a provable (or at least strongly believed) best-case of O(nlogn), and that we already have algorithms that meet that. If that is correct, most improvements would probably come from hardware and software use-case tuning.

Big-O notation is a theoretical tool, it's somewhat useful in practice but it won't necessarily tell you which of two algorithms is faster. It doesn't tell you about cache performance, memory requirements, or even if there's a large coefficient on that n * log(n) term.
Post reply on HN