Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

71–80 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#71

Post by gasping »

zobzu wrote: post #43

Heres a probably unpopular opinion.... Pipes make things a bit slow. A native pipeless program would be a good bit faster - incl. an acid db. Note that doing this in python and expecting it to beat grep wont work... The other thing is that hadoop - and some others are slow on big data (peta, or more) vs own tools. Theyre necessary/used because of massive clustering (10x the hardware deployed easily beats making ur ow…

> incl. an acid db.

Why would you want to use a database for this problem? The input data would take time to load into an ACID db and we're only interested in a single ternary value within that data. The output data is just a few lists of boolean values so it has no reason to be in a database either.

This is a textbook stream processing problem. Adding a database creates more complexity for literally no benefit assuming the requirements in the linked article were complete. I would be baffled to see a solution to this problem that was anything more than a stream processor, to say nothing of a database being involved.

Re: Command-line tools can be faster than your Hadoop cluster

#73

Post by aadrake »

Hi all, original author here.

Some have questioned why I would spend the time advocating against the use of Hadoop for such small data processing tasks as that's clearly not when it should be used anyway. Sadly, Big Data (tm) frameworks are often recommended, required, or used more often than they should be. I know to many of us it seems crazy, but it's true. The worst I've seen was Hadoop used for a processing task of less than 1MB. Seriously.

Also, much agreement with those saying there should be more education effort when it comes to teaching command line tools. O'Reilly even has a book out on the topic: http://shop.oreilly.com/product/0636920032823.do

Thank you for all the comments and support.

Re: Command-line tools can be faster than your Hadoop cluster

#74

Post by TylerE »

hanoz wrote: post #63
danso wrote: post #11

I'm becoming a stronger and stronger advocate of teaching command-line interfaces to even programmers at the novice level...it's easier in many ways to think of how data is being worked on by "filters" and "pipes"...and more importantly, every time you try a step, something happens ...making it much easier to interactively iterate through a process. That it also happens to very fast and powerful (when memory isn't a…

Your CSV peeking epiphany was in essence a matter of code vs. tools though rather than necessarily CLI vs. GUI. On Windows you might just as well have discovered you could fire up Linqpad and enter File.ReadLines("massive.csv").First() for example.

Or just use vim or any other editor smart enough not to try to slurp the whole file in one go.

Re: Command-line tools can be faster than your Hadoop cluster

#75

Post by aadrake »

haddr wrote: post #40

great article! PS. probably some hardcore unix guy would tell you that you are abusing cat. The first cat can be avoided, and you might gain even better performance. Also using gnu grep seems to be faster.

Thank you for the compliment. Point taken on cat, but that's the way I like to introduce the process to people. I took cat and grep out at the end of the article anyway.

Re: Command-line tools can be faster than your Hadoop cluster

#76

Post by jacquesm »

rhizome wrote: post #46
underpants wrote: post #25

Earlier quoted context omitted.

In a similar situation. In fact stupider. We have a 120Gb baseline of data inside a relational store. The vendor has a file stream option that allows blobs to be stored on disk instead of the transaction log and be pushed through the DB server rather than using our current CIFS DFS cluster. So lets stick our 950Gb static document load in there too (while i was on holiday typically) and off we go. Do thing starts goin…

Resume-Driven Development

Being in charge of large budgets is better for CEOs and CTOs than being in charge of small budgets. It makes you look more impressive so they go for it like lemmings over the cliff, sometimes taking entire companies with them.

Re: Command-line tools can be faster than your Hadoop cluster

#77

Post by nathancahill »

dalke wrote: post #27
EpicEng wrote: post #10

Earlier quoted context omitted.

Exactly this just happened where I work. The CIO was recommending Hadoop on AWS for our image processing/analysis jobs. We process a single set of images at a time which come in around ~1.5GB. The output data size is about 1.2GB. Not a good candidate for Hadoop but, you know... "big data", right?

Indeed. For more real-world examples of people who thought that they had "big data", but didn't, see https://news.ycombinator.com/item?id=6398650 ("Don't use Hadoop - your data isn't that big"). The linked essay has: > They handed me a flash drive with all 600MB of their data on it (not a sample, everything). For reasons I can't understand, they were unhappy when my solution involved pandas.read_csv rather than Hadoo…

"Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it." -Dan Ariely

Re: Command-line tools can be faster than your Hadoop cluster

#78

Post by make3 »

MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

you know, or the real world reasonnable mature engineering answer, a Java/C#/C++ scalable parallel tool using modern libraries and MPI if it ever needs to scale.

Re: Command-line tools can be faster than your Hadoop cluster

#80

Post by Spooky23 »

danso wrote: post #11

I'm becoming a stronger and stronger advocate of teaching command-line interfaces to even programmers at the novice level...it's easier in many ways to think of how data is being worked on by "filters" and "pipes"...and more importantly, every time you try a step, something happens ...making it much easier to interactively iterate through a process. That it also happens to very fast and powerful (when memory isn't a…

A few years ago between projects, my coworkers cooked up some satirical amazing Web 2.0 data science tools. They used git, did a screencast and distributed it internally.

It was basically a few compiled perl scripts and some obfuscated shell scripts with a layer of glitz. People actually used it and LOVED it... It was supposedly better than the real tools some groups were using.

It was one of the more epic work trolls I've ever seen!

Post reply on HN