Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

1–10 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#2

Post by greenyoda »

Shell commands are great for data processing pipelines because you get parallelism for free. For proof, try a simple example in your terminal.

    sleep 3 | echo "Hello world."
That doesn't really prove anything about data processing pipelines, since echo "Hello world." doesn't need to wait for any input from the other process; it can run as soon as the process is forked.

    cat *.pgn | grep "Result" | sort | uniq -c
Does this have any advantage over the more straightforward verson below?

    grep -h "Result" *.pgn | sort | uniq -c
Either the cat process or the grep process is going to be waiting for disk I/Os to complete before any of the later processes have data to work on, so splitting it into two processes doesn't seem to buy you any additional concurrency. You would, however, be spending extra time in the kernel to execute the read() and write() system calls to do the interprocess communication on the pipe between cat and grep.

Also, the parallelism of a data processing pipeline is going to be constrained by the speed of the slowest process in it: all the processes after it are going to be idle while waiting for the slow process to produce output, and all the processes before it are going to be idle once the slow process has filled its pipe's input buffers. So if one of the processes in the pipeline takes 100 times as long as the other three, Amdahl's Law[1] suggests that you won't get a big win from breaking it up into multiple processes.

[1] https://en.wikipedia.org/wiki/Amdahl%27s_law

Edit: As someone pointed out, my example needed "grep -h". Fixed.

Re: Command-line tools can be faster than your Hadoop cluster

#4

Post by petemc_ »

greenyoda wrote: post #2

Shell commands are great for data processing pipelines because you get parallelism for free. For proof, try a simple example in your terminal. sleep 3 | echo "Hello world." That doesn't really prove anything about data processing pipelines, since echo "Hello world." doesn't need to wait for any input from the other process; it can run as soon as the process is forked. cat *.pgn | grep "Result" | sort | uniq -c Does t…

You could skip both cat and grep and do it all in awk. Also if speed was an issue you would want to make sure LANG=C is set for grep.

Edit: I see they did use awk later in article, I should really read all of things before commenting.

Re: Command-line tools can be faster than your Hadoop cluster

#6

Post by simonster »

sgt101 wrote: post #3

on a couple of GB this is true, actually if you have ssd's I'd expect any non compute bound task to be faster on a single machine up to ~10gb after which the disk parallelism should kick in and Hadoop should start to win.

If you want disk parallelism, RAID 0 is probably easier than Hadoop.

Re: Command-line tools can be faster than your Hadoop cluster

#7

Post by crcsmnky »

Perhaps I'm missing something. It appears that the author is recommending against using Hadoop (and related tools) for processing 3.5GB of data. Who in the world thought that would be a good idea to begin with?

The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they can sound intelligent with a certain segment of the population.

That said, I like seeing how people put together their own CLI-based processing pipelines.

Re: Command-line tools can be faster than your Hadoop cluster

#8

Post by omoikane »

greenyoda wrote: post #2

Shell commands are great for data processing pipelines because you get parallelism for free. For proof, try a simple example in your terminal. sleep 3 | echo "Hello world." That doesn't really prove anything about data processing pipelines, since echo "Hello world." doesn't need to wait for any input from the other process; it can run as soon as the process is forked. cat *.pgn | grep "Result" | sort | uniq -c Does t…

"grep " is not the same as "cat | grep ", in that the former will prefix lines with filenames if there is more than one input file. What you want instead is "grep -h ".

The advantage of using cat, therefore, is the few seconds of laziness saved in not reading the manual.

Re: Command-line tools can be faster than your Hadoop cluster

#9

Post by clarkm »

greenyoda wrote: post #2

Shell commands are great for data processing pipelines because you get parallelism for free. For proof, try a simple example in your terminal. sleep 3 | echo "Hello world." That doesn't really prove anything about data processing pipelines, since echo "Hello world." doesn't need to wait for any input from the other process; it can run as soon as the process is forked. cat *.pgn | grep "Result" | sort | uniq -c Does t…

You can make it even faster by using fgrep since you're not searching for a regex.

Re: Command-line tools can be faster than your Hadoop cluster

#10

Post by EpicEng »

crcsmnky wrote: post #7

Perhaps I'm missing something. It appears that the author is recommending against using Hadoop (and related tools) for processing 3.5GB of data. Who in the world thought that would be a good idea to begin with? The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they…

Exactly this just happened where I work. The CIO was recommending Hadoop on AWS for our image processing/analysis jobs. We process a single set of images at a time which come in around ~1.5GB. The output data size is about 1.2GB. Not a good candidate for Hadoop but, you know... "big data", right?
Post reply on HN