Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

151–160 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#151

Post by nine_k »

AnthonyMouse wrote: post #106
threeseed wrote: post #90

Earlier quoted context omitted.

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

The Unix people have thought of these things. You can easily do them with command line tools. > Operational Supportability: How do you monitor the operation ? Downloading files with wget will create files and directories as it proceeds. You can observe and count them to determine progress, or pass a shell script to xargs that writes whatever progress data you like to a file before/after calling wget. > Restart Recove…

Unix tools are composable. Functional languages (e.g. Clojure) are all about composability. While bash might be a reasonable glue language, I wonder why Clojure wouldn't be — and it could probably be as compact, if not terser.

The problem of the Hadoop approach is that the overhead of parallelization over multiple hosts is serious, and the task fits one machine neatly. A few GBs of data can and should be processed on one node; Hadoop is for terabytes.

Re: Command-line tools can be faster than your Hadoop cluster

#152

Post by recursive »

alrs wrote: post #105
hanoz wrote: post #63

Earlier quoted context omitted.

Your CSV peeking epiphany was in essence a matter of code vs. tools though rather than necessarily CLI vs. GUI. On Windows you might just as well have discovered you could fire up Linqpad and enter File.ReadLines("massive.csv").First() for example.

The example was a multi-gigabyte CSV file. You just sucked the whole thing off the disk into RAM so that you could shave off the first line. If you're unlucky, you started swapping out to disk about halfway through.

That code you're replying about was carefully and correctly written. You just replied as if you know how it works just so you could look like you know what you're talking about.

If you're unlucky, someone who actually knows how File.ReadLines() works will show up in an hour or two and explain that it's lazily evaluated.

Re: Command-line tools can be faster than your Hadoop cluster

#153

Post by slagfart »

Symbiote wrote: post #113
threeseed wrote: post #90

Earlier quoted context omitted.

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

'pv' shows a progress bar, something like '&& touch $x.success' can help restart recovery. I'd probably pick the shell approach for something I expect to be a one-off, but reconsider each time the task is repeated. I printed http://xkcd.com/1205/ and stuck it to the wall. It's a useful reference when someone seems to be under or overengineering something.

That chart assumes 24 hours days. The reality of (my?) productivity is that I have perhaps six productive hours in a day. If I can save eight productive hours per month, that's sixteen days a year, not four.

Re: Command-line tools can be faster than your Hadoop cluster

#154

Post by kansface »

freehunter wrote: post #149
developer1 wrote: post #140

Earlier quoted context omitted.

The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example, but it's very common for shell scripts to have horrible error handling and checks for important things. The shell is just not suitable for extremely robust programs. I would bet that 80%+ of people who think they're good at shell scripting... aren't.

Yeah but 80% of the people writing Java and think they're good aren't as well. And plenty of companies support Java. The answer isn't "don't use it", it's "train your programmers in the languages they use".

The sorts of bugs people experience with Java mostly result in a crashed/stalled/hung process. Bash bugs erase your entire file system. The thing about Bash is that it is trivially easy to make these sorts of mistakes- the language just isn't suitable to general purpose scripting.

Re: Command-line tools can be faster than your Hadoop cluster

#155

Post by mikeash »

volker48 wrote: post #15
EpicEng wrote: post #10

Earlier quoted context omitted.

Exactly this just happened where I work. The CIO was recommending Hadoop on AWS for our image processing/analysis jobs. We process a single set of images at a time which come in around ~1.5GB. The output data size is about 1.2GB. Not a good candidate for Hadoop but, you know... "big data", right?

If the data can fit on a thumb drive it's not big data.

Would it be fair to approximate it as, "if you can lift it, it's not big data"?

Re: Command-line tools can be faster than your Hadoop cluster

#156

Post by Blackthorn »

Swizec wrote: post #41

Maybe I come from a weird world, or even a weird generation. But when I was in high school, Linux fanboyism was at its peak and just like people get all wound up on bands and such, us geeks got wound up on open-source and linux and fck Micro$oft etc. etc. This was early-ish 2000's. As a result. Every serious programmer I know, especially those who are about my age, lives their life in the CLI. It always comes a surpr…

Can confirm, was there in high school at the same time and had the same experience :-)

I still use programming languages like AWK to this day (I even list it as a known language on my resume; someone commented on it once, it's always a +1 for the company if they do). Recently though I've been exposed to some stuff in the Windows world that makes me think that, sure, they evolved slower but they evolved in a really interesting direction and possibly pulled themselves out of the local maxima that is the Unix world. (I'm referring to PowerShell.)

It's a little weird to me when devs don't immediately go to the command line, given that's just how I learned, but you've gotta recognize that it's not a flaw that they don't. Everyone just learned a little differently.

Re: Command-line tools can be faster than your Hadoop cluster

#157

Post by na85 »

threeseed wrote: post #90
MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

I agree, it's much better to re-invent the wheel.

Re: Command-line tools can be faster than your Hadoop cluster

#158

Post by philsnow »

salgernon wrote: post #18
omoikane wrote: post #8

Earlier quoted context omitted.

"grep " is not the same as "cat | grep ", in that the former will prefix lines with filenames if there is more than one input file. What you want instead is "grep -h ". The advantage of using cat, therefore, is the few seconds of laziness saved in not reading the manual.

The advantage to using "cat foo | grep pattern" is that it is trivial to ^p and edit the pattern before adding the next pipeline sequence.

fwiw

    $
no shell I'm aware of restricts you to placing redirections at the end, you can throw them on the beginning no problem.

Re: Command-line tools can be faster than your Hadoop cluster

#159

Post by eru »

nine_k wrote: post #151
AnthonyMouse wrote: post #106

Earlier quoted context omitted.

The Unix people have thought of these things. You can easily do them with command line tools. > Operational Supportability: How do you monitor the operation ? Downloading files with wget will create files and directories as it proceeds. You can observe and count them to determine progress, or pass a shell script to xargs that writes whatever progress data you like to a file before/after calling wget. > Restart Recove…

Unix tools are composable. Functional languages (e.g. Clojure) are all about composability. While bash might be a reasonable glue language, I wonder why Clojure wouldn't be — and it could probably be as compact, if not terser. The problem of the Hadoop approach is that the overhead of parallelization over multiple hosts is serious, and the task fits one machine neatly. A few GBs of data can and should be processed on…

> [...] I wonder why Clojure wouldn't be — and it could probably be as compact, if not terser.

Because Clojure is a goo language, that question depends mostly on the libraries available for Clojure.

(Whereas some other languages are worse at gluing, so libraries will only help you so far.)

Re: Command-line tools can be faster than your Hadoop cluster

#160

Post by eru »

jessaustin wrote: post #84
make3 wrote: post #78

Earlier quoted context omitted.

you know, or the real world reasonnable mature engineering answer, a Java/C#/C++ scalable parallel tool using modern libraries and MPI if it ever needs to scale.

Yeah if you're Google. Most people are not, and wget is plenty. After all it's written in C.

Or use curl, for a slightly better engineered wget.
Post reply on HN