Command-line tools can be faster than your Hadoop cluster

Post by mappu » Mon, Jan 19, 2015, 12:43 AM UTC

m_mueller wrote: post #85

mabbo wrote: post #48

I had an intern over the summer, working on a basic A/B Testing framework for our application (a very simple industrial handscanner tool used inside warehouses by a few thousand employees). When we came to the last stage, analysis, he was keen to use MapReduce so we let him. In the end though, his analysis didn't work well, took ages to process when it did, and didn't provide the answers we needed. The code wasn't ma…

On a tangent, I'd be interested in how you format heavily piped bash code for documentation. Can comments be intersparsed there?

    # @param $1 whatever
    chess_extract_scores() {
         awk blah blah blah
    }

    cat foo | grep bar | chess_extract_scores

Post by pmoriarty » Mon, Jan 19, 2015, 1:00 AM UTC

sam_lowry_ wrote: post #45

Next to using `xargs -P 8 -n 1` to parallellize jobs locally, take a look at paexec, GNU parallel replacement that just works. See https://github.com/cheusov/paexec

Post by angrybits » Mon, Jan 19, 2015, 1:05 AM UTC

threeseed wrote: post #104

x0x0 wrote: post #102

Earlier quoted context omitted.

Sure -- you can level multiple complaints. What about the failed pages? How about shoving those on a queue and retrying n times with an exponential backoff between. What about the total number of failed pages? What about failed pages by site? etc etc etc But so what -- the principle is still sound. All I described is still a 100 line python script, written in an afternoon, instead of 3 weeks of working bringing up em…

Anybody can throw some crap together and make it stick. And it's a perfectly valid solution. My issue is when there is criticism laid against those solutions which are actually engineered in a way that allows for supportability and extensibility. They are arguably far more important than execution time.

Post by nextos » Mon, Jan 19, 2015, 1:12 AM UTC

sauere wrote: post #115

AnthonyMouse wrote: post #106

Earlier quoted context omitted.

The Unix people have thought of these things. You can easily do them with command line tools. > Operational Supportability: How do you monitor the operation ? Downloading files with wget will create files and directories as it proceeds. You can observe and count them to determine progress, or pass a shell script to xargs that writes whatever progress data you like to a file before/after calling wget. > Restart Recove…

I could not agree more. And even with the things you mentioned, such a script will still be tiny and very readable. You just have to love the simplicity.

Post by coderdude » Mon, Jan 19, 2015, 1:16 AM UTC

AnthonyMouse wrote: post #106

threeseed wrote: post #90

Earlier quoted context omitted.

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

The Unix people have thought of these things. You can easily do them with command line tools. > Operational Supportability: How do you monitor the operation ? Downloading files with wget will create files and directories as it proceeds. You can observe and count them to determine progress, or pass a shell script to xargs that writes whatever progress data you like to a file before/after calling wget. > Restart Recove…