Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

81–90 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#81

Post by azernik »

EpicEng wrote: post #10
crcsmnky wrote: post #7

Perhaps I'm missing something. It appears that the author is recommending against using Hadoop (and related tools) for processing 3.5GB of data. Who in the world thought that would be a good idea to begin with? The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they…

Exactly this just happened where I work. The CIO was recommending Hadoop on AWS for our image processing/analysis jobs. We process a single set of images at a time which come in around ~1.5GB. The output data size is about 1.2GB. Not a good candidate for Hadoop but, you know... "big data", right?

Perhaps a good way of explaining it to management is, "If it fits on a smartphone SD card, it's tiny data. If it fits on a laptop hard drive, it's maybe medium data." I think at that point many of these conversations would end.

Re: Command-line tools can be faster than your Hadoop cluster

#82

Post by aadrake »

greenyoda wrote: post #2

Shell commands are great for data processing pipelines because you get parallelism for free. For proof, try a simple example in your terminal. sleep 3 | echo "Hello world." That doesn't really prove anything about data processing pipelines, since echo "Hello world." doesn't need to wait for any input from the other process; it can run as soon as the process is forked. cat *.pgn | grep "Result" | sort | uniq -c Does t…

Thank you for the comment. The fact that it can run independently as soon as it's forked was exactly the point I was making.

Also, the cat | grep pipeline is illustrative. I remove it at the end.

Re: Command-line tools can be faster than your Hadoop cluster

#83

Post by threeseed »

EpicEng wrote: post #10
crcsmnky wrote: post #7

Perhaps I'm missing something. It appears that the author is recommending against using Hadoop (and related tools) for processing 3.5GB of data. Who in the world thought that would be a good idea to begin with? The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they…

Exactly this just happened where I work. The CIO was recommending Hadoop on AWS for our image processing/analysis jobs. We process a single set of images at a time which come in around ~1.5GB. The output data size is about 1.2GB. Not a good candidate for Hadoop but, you know... "big data", right?

Another explanation is that your CIO is not an idiot but rather they know about future projects that you don't. CIOs want to build capabilities (skills and technologies) not just one off implementations every time.

Not saying this is the case but CIO bashing is all too easy when you're an engineer.

Re: Command-line tools can be faster than your Hadoop cluster

#84

Post by jessaustin »

make3 wrote: post #78
MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

you know, or the real world reasonnable mature engineering answer, a Java/C#/C++ scalable parallel tool using modern libraries and MPI if it ever needs to scale.

Yeah if you're Google. Most people are not, and wget is plenty. After all it's written in C.

Re: Command-line tools can be faster than your Hadoop cluster

#85

Post by m_mueller »

mabbo wrote: post #48

I had an intern over the summer, working on a basic A/B Testing framework for our application (a very simple industrial handscanner tool used inside warehouses by a few thousand employees). When we came to the last stage, analysis, he was keen to use MapReduce so we let him. In the end though, his analysis didn't work well, took ages to process when it did, and didn't provide the answers we needed. The code wasn't ma…

On a tangent, I'd be interested in how you format heavily piped bash code for documentation. Can comments be intersparsed there?

Re: Command-line tools can be faster than your Hadoop cluster

#86

Post by threeseed »

dschiptsov wrote: post #55

Everyone with basic knowledge of CS could realize that Hadoop is a waste. Unfortunately, it isn't about efficiency at all. It just memeization. Bigdata? Hadoop! Runs everywhere. Same BS like Webscale? MongoDB! meme.

Well sorry but you don't have a clue what you're talking about.

I very much work in "big data" with about 2 terabytes of new data coming in every day that has to be ingested and processed with hundreds of jobs running against them. The data needs to be queryable via an SQL like language and analyzed by a dozen data scientists using R or Map Reduce.

There isn't anything on the market today that has been proven to work in environments like this and has the tooling to back it up. Unless you want to prove everyone e.g. Netflix, Linkedin, Spotify, Apple, Microsoft wrong ?

Re: Command-line tools can be faster than your Hadoop cluster

#87

Post by untog »

make3 wrote: post #78
MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

you know, or the real world reasonnable mature engineering answer, a Java/C#/C++ scalable parallel tool using modern libraries and MPI if it ever needs to scale.

You usually don't need that though, that's the point. If you're building an entire service around page crawling, sure. If you're doing a one-off task, don't bother.

Re: Command-line tools can be faster than your Hadoop cluster

#88

Post by clarkm »

simpleigh wrote: post #19
clarkm wrote: post #9

Earlier quoted context omitted.

You can make it even faster by using fgrep since you're not searching for a regex.

The article addresses this: > The -F for grep indicates that we are only matching on fixed strings and not doing any fancy regex, and can offer a small speedup, which I did not notice in my testing. I guess grep is probably clever enough to choose a faster matching algorithm once it's parsed the pattern and discovered it doesn't contain any regex fun.

In general grep seems smart enough that it would do that, but it hasn't been my experience. Just last week I was searching through a couple hundred gigs small xml files. I found that:

    $ LC_ALL=C fgrep -r STRING .
was much faster than plain grep. This was on a CentOS 5 box, so maybe newer versions of grep are smarter.

But then again, if I was on a newer box I'd just install and use ack or ag.

Re: Command-line tools can be faster than your Hadoop cluster

#89

Post by raincom »

Hadoop is replacing many datawarehousing dbs like netezza, teradata, exadata. In the process, many datwarehousing developers have become hadoop developers, who write sql code; after all, hadoop got a sql interface via hive.

Informatica (another ETL tool) also provides another tool called powerexchange, which automatically generates MR code for hadoop.

Whenever you hear hadoop, first ask yourself whether it is another disguised datawarehousing stuff.

Re: Command-line tools can be faster than your Hadoop cluster

#90

Post by threeseed »

MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

Oh right the "cool kids" approach.

Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform easily to do X, Y, Z after the crawling ?

I can't stand developers who come up with the xargs/wget approach, hack something together and then walk away from it. I've seen it far too often and it's great for the short term. Dreadful for the long term.

Post reply on HN