Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

91–100 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#91

Post by MrBuddyCasino »

make3 wrote: post #78
MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

you know, or the real world reasonnable mature engineering answer, a Java/C#/C++ scalable parallel tool using modern libraries and MPI if it ever needs to scale.

That would be my first gut reaction too, but if it is as simple as downloading webpages, this is actually a really great solution. I suspect he used that when he built Milo, a now defunct startup sold to eBay where they had to update prices and inventory data regularly. A startup should make different choices than Google.

Re: Command-line tools can be faster than your Hadoop cluster

#93

Post by UrMomReadsHN »

Swizec wrote: post #41

Maybe I come from a weird world, or even a weird generation. But when I was in high school, Linux fanboyism was at its peak and just like people get all wound up on bands and such, us geeks got wound up on open-source and linux and fck Micro$oft etc. etc. This was early-ish 2000's. As a result. Every serious programmer I know, especially those who are about my age, lives their life in the CLI. It always comes a surpr…

15 years ago when i first started programming i learned on the *nix CLI. I worked that way for years, thats just how it was done. Well, I started using an IDE in windows and now.... I really like it. I don't want to go back to the command line. That being said, I still use the Windows command line from time to time.

Re: Command-line tools can be faster than your Hadoop cluster

#94

Post by wglb »

Swizec wrote: post #41

Maybe I come from a weird world, or even a weird generation. But when I was in high school, Linux fanboyism was at its peak and just like people get all wound up on bands and such, us geeks got wound up on open-source and linux and fck Micro$oft etc. etc. This was early-ish 2000's. As a result. Every serious programmer I know, especially those who are about my age, lives their life in the CLI. It always comes a surpr…

Every serious programmer I know, especially those who are about my age, lives their life in the CLI.

This is not even close to true. Of the developers that I know, many of them quite serious, live in the Windows world, and are very happy with for example VS or eclipse. This is very likely to be the case in most BFEs.

Re: Command-line tools can be faster than your Hadoop cluster

#95

Post by mc808 »

hueving wrote: post #68
hanoz wrote: post #63

Earlier quoted context omitted.

Your CSV peeking epiphany was in essence a matter of code vs. tools though rather than necessarily CLI vs. GUI. On Windows you might just as well have discovered you could fire up Linqpad and enter File.ReadLines("massive.csv").First() for example.

Do you not see the horrific syntax of what you just suggested as simple?

It's pretty clear what it does. It's also C#, so building up to a less trivial task will be much less horrific than

find . -type f -name '*.pgn' -print0 | xargs -0 -n4 -P4 mawk '/Result/ { split($0, a, "-"); res = substr(a[1], length(a[1]), 1); if (res == 1) white++; if (res == 0) black++; if (res == 2) draw++ } END { print white+black+draw, white, black, draw }' | mawk '{games += $1; white += $2; black += $3; draw += $4; } END { print games, white, black, draw }'

Re: Command-line tools can be faster than your Hadoop cluster

#96

Post by sabalaba »

I've had the pleasure and displeasure of working with small datasets (~7.5GB of images) in shell. One often needs to send SIGINT to the shell when it starts to glob expand or tab complete a folder with millions of files. But besides minor issues like that, command line tools get the job done.

Re: Command-line tools can be faster than your Hadoop cluster

#97

Post by rkwasny »

Bottom line is - you do not need hadoop until you cross 2TB of data to be processed (uncompressed). Modern servers ( bare metal ones, not what AWS sells you ) are REALLY FAST and can crunch massive amounts of data.

Just use a proper tools, well optimized code written in C/C++/Go/etc - not all the crappy JAVA framework-in-a-framework^N architecture that abstracts thinking about the CPU speed.

Bottom line, the popular saying is true: "Hadoop is about writing crappy code and then running it on a massive scale."

Re: Command-line tools can be faster than your Hadoop cluster

#98

Post by wglb »

prodigal_erik wrote: post #54
Swizec wrote: post #41

Maybe I come from a weird world, or even a weird generation. But when I was in high school, Linux fanboyism was at its peak and just like people get all wound up on bands and such, us geeks got wound up on open-source and linux and fck Micro$oft etc. etc. This was early-ish 2000's. As a result. Every serious programmer I know, especially those who are about my age, lives their life in the CLI. It always comes a surpr…

People are always surprised when I mention that the Microsoft devs I worked with had free access to the highest tiers of Visual Studio, yet what they actually worked in was vim and the internal fork of make. I don't know whether that's still true; it's been a decade now.

When you say "Microsoft Devs", do you mean MS employees?

If so I heard s similar tale back in the first days of .net, where most of the guys in one particular group were using emacs.

Re: Command-line tools can be faster than your Hadoop cluster

#99

Post by treve »

rkwasny wrote: post #97

Bottom line is - you do not need hadoop until you cross 2TB of data to be processed (uncompressed). Modern servers ( bare metal ones, not what AWS sells you ) are REALLY FAST and can crunch massive amounts of data. Just use a proper tools, well optimized code written in C/C++/Go/etc - not all the crappy JAVA framework-in-a-framework^N architecture that abstracts thinking about the CPU speed. Bottom line, the popular…

What makes 2TB the cutoff?

Re: Command-line tools can be faster than your Hadoop cluster

#100

Post by ronreiter »

Hadoop is highly inefficient when using default MapReduce configuration. And a single Macbook Pro machine is much stronger than 7 c1.medium instances.

Bottom line - run the same thing over Apache Tez with a cluster that has the same computational resources as your laptop, and I'm pretty sure you'll see the same results.

Post reply on HN