Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

11–20 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#11

Post by danso »

I'm becoming a stronger and stronger advocate of teaching command-line interfaces to even programmers at the novice level...it's easier in many ways to think of how data is being worked on by "filters" and "pipes"...and more importantly, every time you try a step, something happens...making it much easier to interactively iterate through a process.

That it also happens to very fast and powerful (when memory isn't a limiting factor) is nice icing on the cake. I moved over to doing much more on CLI after realizing that doing something as simple as "head -n 1 massive.csv" to inspect headers of corrupt multi-gb CSV files made my data-munging life substantially more enjoyable than opening them up in Sublime Text.

Re: Command-line tools can be faster than your Hadoop cluster

#12

Post by cmurf »

simonster wrote: post #6
sgt101 wrote: post #3

on a couple of GB this is true, actually if you have ssd's I'd expect any non compute bound task to be faster on a single machine up to ~10gb after which the disk parallelism should kick in and Hadoop should start to win.

If you want disk parallelism, RAID 0 is probably easier than Hadoop.

That would depend on the data set and the strip size. Striping is good for streaming. Linear/concat of 2+ drives with XFS would be faster with a lot of files than end up in separate AG's on separate drives which can be accessed in parallel.

Re: Command-line tools can be faster than your Hadoop cluster

#13

Post by texthompson »

EpicEng wrote: post #10
crcsmnky wrote: post #7

Perhaps I'm missing something. It appears that the author is recommending against using Hadoop (and related tools) for processing 3.5GB of data. Who in the world thought that would be a good idea to begin with? The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they…

Exactly this just happened where I work. The CIO was recommending Hadoop on AWS for our image processing/analysis jobs. We process a single set of images at a time which come in around ~1.5GB. The output data size is about 1.2GB. Not a good candidate for Hadoop but, you know... "big data", right?

That's not even medium data. Most people probably would be surprised to find out that their data could be stored and processed on an iPhone, and that using heavier duty tools isn't necessary or worthwhile.

Re: Command-line tools can be faster than your Hadoop cluster

#14

Post by KaiserPro »

sgt101 wrote: post #3

on a couple of GB this is true, actually if you have ssd's I'd expect any non compute bound task to be faster on a single machine up to ~10gb after which the disk parallelism should kick in and Hadoop should start to win.

Depends on the dick, depends on the storage.

HDFS is a psudeo block interface. If you have a real filesystem like lustre, or GPFS, not only do you have the abilty to use other tools, you can use that storage for other things.

In the case of GPFS, you have configurable redundancy. Sadly with lustre, you need decent hardware, otherwise you're going to loose data.

In all these things, paying bottom dollar for hardware, forgoing support is a false economy. At scales of 1pb+ (which is about 1/2 a rack now) its much much cheaper to use off the shelf parts with 24/7 support than "softwareing" your way out.

Re: Command-line tools can be faster than your Hadoop cluster

#15

Post by volker48 »

EpicEng wrote: post #10
crcsmnky wrote: post #7

Perhaps I'm missing something. It appears that the author is recommending against using Hadoop (and related tools) for processing 3.5GB of data. Who in the world thought that would be a good idea to begin with? The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they…

Exactly this just happened where I work. The CIO was recommending Hadoop on AWS for our image processing/analysis jobs. We process a single set of images at a time which come in around ~1.5GB. The output data size is about 1.2GB. Not a good candidate for Hadoop but, you know... "big data", right?

If the data can fit on a thumb drive it's not big data.

Re: Command-line tools can be faster than your Hadoop cluster

#16

Post by wallflower »

Awk and Sed aren't very accessible to most people who did not grow up learning those tools.

The whole point of tools built on top of Hadoop (Hive/Pig/HBase) is to make large scale data processing more accessible (by hiding the map-reduce as much as possible). Not everyone will want to write a Java map-reduce in Hadoop. However, many can write a HiveQL statement or Pig textual script. Amazon Redshift brings it even farther - they are a Postgres compatible database, meaning you can connect your Crystal Reports/Tableau data analysis tool to it, treating it like a traditional SQL database.

Re: Command-line tools can be faster than your Hadoop cluster

#17

Post by NyxWulf »

This also isn't a straight either or proposition. I build local command line pipelines and do testing and/or processing. When either the amount of data needed to be processed passes into the range where memory or network bandwidth makes the processing more efficient on a Hadoop cluster I make some fairly minimal conversions and run the stream processing on the Hadoop cluster in streaming mode. It hasn't been uncommon for my jobs to be much faster than the same jobs run on the cluster with Hive or some other framework. Much of the speed boils down to the optimizer and the planner.

Overall I find it very efficient to use the same toolset locally and then scale it up to a cluster when and if I need to.

Re: Command-line tools can be faster than your Hadoop cluster

#18

Post by salgernon »

omoikane wrote: post #8
greenyoda wrote: post #2

Shell commands are great for data processing pipelines because you get parallelism for free. For proof, try a simple example in your terminal. sleep 3 | echo "Hello world." That doesn't really prove anything about data processing pipelines, since echo "Hello world." doesn't need to wait for any input from the other process; it can run as soon as the process is forked. cat *.pgn | grep "Result" | sort | uniq -c Does t…

"grep " is not the same as "cat | grep ", in that the former will prefix lines with filenames if there is more than one input file. What you want instead is "grep -h ". The advantage of using cat, therefore, is the few seconds of laziness saved in not reading the manual.

The advantage to using "cat foo | grep pattern" is that it is trivial to ^p and edit the pattern before adding the next pipeline sequence.

Re: Command-line tools can be faster than your Hadoop cluster

#19

Post by simpleigh »

clarkm wrote: post #9
greenyoda wrote: post #2

Shell commands are great for data processing pipelines because you get parallelism for free. For proof, try a simple example in your terminal. sleep 3 | echo "Hello world." That doesn't really prove anything about data processing pipelines, since echo "Hello world." doesn't need to wait for any input from the other process; it can run as soon as the process is forked. cat *.pgn | grep "Result" | sort | uniq -c Does t…

You can make it even faster by using fgrep since you're not searching for a regex.

The article addresses this:

> The -F for grep indicates that we are only matching on fixed strings and not doing any fancy regex, and can offer a small speedup, which I did not notice in my testing.

I guess grep is probably clever enough to choose a faster matching algorithm once it's parsed the pattern and discovered it doesn't contain any regex fun.

Post reply on HN