tl;dr: You do not have a big data problem.
I would have said Right tool for the job.
Command-line tools can be faster than your Hadoop cluster
31–40 of 315 posts
Re: Command-line tools can be faster than your Hadoop cluster
#32until ~10 GB you'd better keep on going with single core machines, you'll see some improvementes with bigger sets > 100 GB
Re: Command-line tools can be faster than your Hadoop cluster
#33remove cat and use grep directly please...
Re: Command-line tools can be faster than your Hadoop cluster
#34Awk and Sed aren't very accessible to most people who did not grow up learning those tools. The whole point of tools built on top of Hadoop (Hive/Pig/HBase) is to make large scale data processing more accessible (by hiding the map-reduce as much as possible). Not everyone will want to write a Java map-reduce in Hadoop. However, many can write a HiveQL statement or Pig textual script. Amazon Redshift brings it even fa…
I think the author's point was that the example in question was orders of magnitude smaller than "big data" and that it was more efficient to process it on a single machine, not that Hadoop and friends aren't easy to use.
Re: Command-line tools can be faster than your Hadoop cluster
#35Perhaps I'm missing something. It appears that the author is recommending against using Hadoop (and related tools) for processing 3.5GB of data. Who in the world thought that would be a good idea to begin with? The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they…
Lots of people think that is "big data". For most people if it's too big for an Excel spreadsheet, it's "big data" and the way you process big data is with Hadoop. Of course once you show them the billable hours difference between setting up a Hadoop cluster, and (in my case at least) using python libraries on a MBP, they change their minds real fast. Its just a matter of "big data" being a new thing, people will figure it out as time goes on and things settle down.
Re: Command-line tools can be faster than your Hadoop cluster
#36I feel ag (silver surfer, a grep-ish alternative) should be mentioned (even though he dropped it in his final awk/mawk commands) as it tends to be much faster than grep, and considering he cites performance throughout.
Re: Command-line tools can be faster than your Hadoop cluster
#37This also isn't a straight either or proposition. I build local command line pipelines and do testing and/or processing. When either the amount of data needed to be processed passes into the range where memory or network bandwidth makes the processing more efficient on a Hadoop cluster I make some fairly minimal conversions and run the stream processing on the Hadoop cluster in streaming mode. It hasn't been uncommon…
What toolset are you using that you can run both locally and on a Hadoop cluster?
Re: Command-line tools can be faster than your Hadoop cluster
#38I think it is unsafe to parallelize grep with xargs as in done in the article, because, beyond delivery order shuffling, the output of the parallel greps could get mixed up (the beginning of a line is by one grep and the end of a line is from a different grep, so, reading line by line afterwards, you get garbled lines).
See https://www.gnu.org/software/parallel/man.html#DIFFERENCES-B...
Re: Command-line tools can be faster than your Hadoop cluster
#39'xargs -n' elicits fond memories of spawning large jobs to my Openmosix cluster ! I miss Openmosix.
Oh! I lost track of it.
Re: Command-line tools can be faster than your Hadoop cluster
#40great article!
PS. probably some hardcore unix guy would tell you that you are abusing cat. The first cat can be avoided, and you might gain even better performance. Also using gnu grep seems to be faster.