Sites down; but, I wanted to add that at the peak of Hadoop hype, an unnamed company asked how I'd tally up occurrences of a value in a 1gb flat file, where the job would need to be done recurringly on multiple files, quickly. Without wanting to overthink it, I suggested a simple script iterating over the file, incrementing the key/values as they occur. Apparently the correct answer was a Hadoop job. Now, given that…
Isn't that exactly what the Hadoop job would have to do anyway? The whole point of Hadoop is to process the data in a single pass, which is what your method does. As long as you iterate over the file without reading the whole thing into memory first.