11–16 of 16 posts
That's some pretty impressive hardware. Must've been pretty fun. Not something just anyone gets to work on. :)
it's hard to infer if the record is being set due to better hardware or better algorithms. I'd like to see benchmark improvements in algorithms not hardware, but I'm a software guy.
I was under the impression (could be remembering something wrong, however) that sorting algorithms had a provable (or at least strongly believed) best-case of O(nlogn), and that we already have algorithms that meet that. If that is correct, most improvements would probably come from hardware and software use-case tuning.
BTW, these are 100 byte records, so 1TB = 10 billion records. I don't know why they measure in bytes, which is nearly meaningless. I can sort a single 1TB record in my head right now. There, I'm done.
Its hard to understand if you are just a programmer, a computer science graduate can easily identify with size in bytes since they represent accurate amount rather than vague billion entries.
In later case you need to specify two variables size of entry and number of entries.
the larger entries become the easier is to sort them, if they get smaller then you can have an O(n) sorting ability.
The 100 byte records is fixed in literature and well understood by practitioners, what you are experiencing is your naivete.
it's hard to infer if the record is being set due to better hardware or better algorithms. I'd like to see benchmark improvements in algorithms not hardware, but I'm a software guy.
I was under the impression (could be remembering something wrong, however) that sorting algorithms had a provable (or at least strongly believed) best-case of O(nlogn), and that we already have algorithms that meet that. If that is correct, most improvements would probably come from hardware and software use-case tuning.
it's hard to infer if the record is being set due to better hardware or better algorithms. I'd like to see benchmark improvements in algorithms not hardware, but I'm a software guy.
It's hardware. They have have 52 servers, each with 16 disks which I assume are capable of I/O at 100MB/s. That's an aggregate disk I/O rate of 83.2 GB/s. Their network switch provides 10Gbps connectivity between each pair, which implies they can move data between nodes at 65 GB/s. With that hardware setup they can read 1TB from disk and distribute in around 15 seconds. They can write sorted data back to disk in 12 s…
it's hard to infer if the record is being set due to better hardware or better algorithms. I'd like to see benchmark improvements in algorithms not hardware, but I'm a software guy.
It's hardware. They have have 52 servers, each with 16 disks which I assume are capable of I/O at 100MB/s. That's an aggregate disk I/O rate of 83.2 GB/s. Their network switch provides 10Gbps connectivity between each pair, which implies they can move data between nodes at 65 GB/s. With that hardware setup they can read 1TB from disk and distribute in around 15 seconds. They can write sorted data back to disk in 12 s…
"We asked ourselves, ‘What does it mean to build a balanced system where we are not wasting any system resources in carrying out high end computation?’” said Vahdat. “If you are idling your processors or not using all your RAM, you’re burning energy and losing efficiency.”
From personal experience, the hardest part of dealing with super-high-end hardware is utilizing the capacity that it gives you. You don't just throw a larger computer at a program written in Blub for a smaller system, and gain instant world-record performance. If nothing else, you usually have to re-architect your code to match the assumptions of the new hardware.
Also, the I/O numbers you're assuming are probably very high: you never get 100% utilization of a network interface -- 70% is more realistic. And for a problem like this, you're bounded not by the product of the bandwidth of the pairwise interconnects (i.e. (10Gbps * 52) / 8 == 65GB/s), but the bandwidth of any single interconnect, because you can get to a point where you're transferring most of the data across a few connections. Also, latency matters a lot, because a small connection latency can easily throttle your overall bandwidth when you're moving lots of small chunks of data.
Realistically, these guys probably had something more like 7Gbps / 8 = ~.88 GB/s bandwidth between nodes, not counting latency. That means that it would take ~1163s to transfer 1TB between any two nodes -- which means that that naive solution (distributed merge sort where the intermediates are transferred between nodes) is already ruled out. So, like I said, we're back to software improvements, if only to utilize the hardware available.