Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

61–70 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#61

Post by MrBuddyCasino »

To quote the memorable Ted Dziuba[0]:

"Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ.

The Taco Bell answer? xargs and wget. In the rare case that you saturate the network connection, add some split and rsync. A "distributed crawler" is really only like 10 lines of shell script."

[0] since his blog is gone: http://readwrite.com/2011/01/22/data-mining-and-taco-bell-pr...

Re: Command-line tools can be faster than your Hadoop cluster

#63

Post by hanoz »

danso wrote: post #11

I'm becoming a stronger and stronger advocate of teaching command-line interfaces to even programmers at the novice level...it's easier in many ways to think of how data is being worked on by "filters" and "pipes"...and more importantly, every time you try a step, something happens ...making it much easier to interactively iterate through a process. That it also happens to very fast and powerful (when memory isn't a…

Your CSV peeking epiphany was in essence a matter of code vs. tools though rather than necessarily CLI vs. GUI. On Windows you might just as well have discovered you could fire up Linqpad and enter File.ReadLines("massive.csv").First() for example.

Re: Command-line tools can be faster than your Hadoop cluster

#64

Post by IndianAstronaut »

x0x0 wrote: post #23
crcsmnky wrote: post #7

Perhaps I'm missing something. It appears that the author is recommending against using Hadoop (and related tools) for processing 3.5GB of data. Who in the world thought that would be a good idea to begin with? The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they…

I've used hadoop at petabyte scale (2+pb input; 10+pb sorted for the job) for machine learning tasks. If you have such a thing on your resume, you will be inundated with employers who have "big data", and at least half will be under 50g with a good chunk of those under 10g. You'll also see multiple (shitty) 16 machine clusters, any of which -- for any task -- could be destroyed by code running on a single decent serv…

Imho, hadoop is only for 100tb or more. Anything less can be easily handled by other tools.

Re: Command-line tools can be faster than your Hadoop cluster

#65

Post by kyllo »

mech4bg wrote: post #53
Swizec wrote: post #41

Maybe I come from a weird world, or even a weird generation. But when I was in high school, Linux fanboyism was at its peak and just like people get all wound up on bands and such, us geeks got wound up on open-source and linux and fck Micro$oft etc. etc. This was early-ish 2000's. As a result. Every serious programmer I know, especially those who are about my age, lives their life in the CLI. It always comes a surpr…

This shouldn't be a surprise. Tons of development is done on Windows. Most game development, obviously Windows app development, .NET websites, etc. There are command line tools there, but in my 10 years of being a Windows developer, GUI tools were more the norm. There's a time and a place for both. Now developing predominantly under Linux, it amazes me how time consuming and clunky some tasks are on the command line…

Windows is just inherently GUI-centric, if you force developers to use Windows they are naturally going to gravitate toward using GUI based tools because Windows command line is so crippled. Tools like Cygwin and Chocolatey are nice in a pinch but they just don't compare to being in a real UNIX environment.

Re: Command-line tools can be faster than your Hadoop cluster

#66

Post by arjie »

azylman wrote: post #37
NyxWulf wrote: post #17

This also isn't a straight either or proposition. I build local command line pipelines and do testing and/or processing. When either the amount of data needed to be processed passes into the range where memory or network bandwidth makes the processing more efficient on a Hadoop cluster I make some fairly minimal conversions and run the stream processing on the Hadoop cluster in streaming mode. It hasn't been uncommon…

What toolset are you using that you can run both locally and on a Hadoop cluster?

I believe he mentioned it. The Hadoop streaming mode.

Re: Command-line tools can be faster than your Hadoop cluster

#67

Post by Someone »

hanoz wrote: post #63
danso wrote: post #11

I'm becoming a stronger and stronger advocate of teaching command-line interfaces to even programmers at the novice level...it's easier in many ways to think of how data is being worked on by "filters" and "pipes"...and more importantly, every time you try a step, something happens ...making it much easier to interactively iterate through a process. That it also happens to very fast and powerful (when memory isn't a…

Your CSV peeking epiphany was in essence a matter of code vs. tools though rather than necessarily CLI vs. GUI. On Windows you might just as well have discovered you could fire up Linqpad and enter File.ReadLines("massive.csv").First() for example.

Running a shell in a GUI doesn't make it lose its "I am a CLI" property. That is a CLI.

Re: Command-line tools can be faster than your Hadoop cluster

#68

Post by hueving »

hanoz wrote: post #63
danso wrote: post #11

I'm becoming a stronger and stronger advocate of teaching command-line interfaces to even programmers at the novice level...it's easier in many ways to think of how data is being worked on by "filters" and "pipes"...and more importantly, every time you try a step, something happens ...making it much easier to interactively iterate through a process. That it also happens to very fast and powerful (when memory isn't a…

Your CSV peeking epiphany was in essence a matter of code vs. tools though rather than necessarily CLI vs. GUI. On Windows you might just as well have discovered you could fire up Linqpad and enter File.ReadLines("massive.csv").First() for example.

Do you not see the horrific syntax of what you just suggested as simple?

Re: Command-line tools can be faster than your Hadoop cluster

#69

Post by mdaniel »

azylman wrote: post #37
NyxWulf wrote: post #17

This also isn't a straight either or proposition. I build local command line pipelines and do testing and/or processing. When either the amount of data needed to be processed passes into the range where memory or network bandwidth makes the processing more efficient on a Hadoop cluster I make some fairly minimal conversions and run the stream processing on the Hadoop cluster in streaming mode. It hasn't been uncommon…

What toolset are you using that you can run both locally and on a Hadoop cluster?

Almost all of them?

The vocabulary of the grandparent comment implies they are using hadoop's streaming mode, and thus one can use a map-reduce streaming abstraction such as MRJob or just plain stdin/stdout; both will work locally and in cluster mode.

Or, if static typing is more agreeable to your development process, running hadoop in "single machine cluster" mode is relatively painless. The same goes for other distributed processing frameworks like Spark.

Re: Command-line tools can be faster than your Hadoop cluster

#70

Post by sedachv »

x0x0 wrote: post #23
crcsmnky wrote: post #7

Perhaps I'm missing something. It appears that the author is recommending against using Hadoop (and related tools) for processing 3.5GB of data. Who in the world thought that would be a good idea to begin with? The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they…

I've used hadoop at petabyte scale (2+pb input; 10+pb sorted for the job) for machine learning tasks. If you have such a thing on your resume, you will be inundated with employers who have "big data", and at least half will be under 50g with a good chunk of those under 10g. You'll also see multiple (shitty) 16 machine clusters, any of which -- for any task -- could be destroyed by code running on a single decent serv…

> Also, hadoop is so painfully slow to develop in it's practically a full employment act for software engineers.

It's comical how bad Hadoop is compared even to the CM Lisp described in Daniel Hillis' PhD dissertation. How do you devolve all the way from that down to "It's like map/reduce. You get one map and one reduce!"

Post reply on HN