Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

131–140 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#131

Post by x0x0 »

angrybits wrote: post #125
threeseed wrote: post #104

Earlier quoted context omitted.

Anybody can throw some crap together and make it stick. And it's a perfectly valid solution. My issue is when there is criticism laid against those solutions which are actually engineered in a way that allows for supportability and extensibility. They are arguably far more important than execution time.

I can't figure out why you're arguing against standard unix tools/idioms in the name of supportability and extensibility? It defies logic.

I think, in many peoples minds, extensibility == pain; either lots of code configuration (hello java, ejb), or xml (hello hadoop, java, spring, ejb), or tons of code (hello java, c++), etc. When nice languages don't make things painful, it sometimes feels like it's wrong, or not really enough work, or in some other way, insufficient. But people can mistake the rituals of programming for getting actual work accomplished.

:shrug: just .02

Re: Command-line tools can be faster than your Hadoop cluster

#132

Post by kevin_thibedeau »

mc808 wrote: post #95
hueving wrote: post #68

Earlier quoted context omitted.

Do you not see the horrific syntax of what you just suggested as simple?

It's pretty clear what it does. It's also C#, so building up to a less trivial task will be much less horrific than find . -type f -name '*.pgn' -print0 | xargs -0 -n4 -P4 mawk '/Result/ { split($0, a, "-"); res = substr(a[1], length(a[1]), 1); if (res == 1) white++; if (res == 0) black++; if (res == 2) draw++ } END { print white+black+draw, white, black, draw }' | mawk '{games += $1; white += $2; black += $3; draw +…

In a real production environment that command line would be put into a script parametrized with named variables and the embedded awk scripts would be changed to here-docs.

Re: Command-line tools can be faster than your Hadoop cluster

#133

Post by peterhunt »

kevin_thibedeau wrote: post #132
mc808 wrote: post #95

Earlier quoted context omitted.

It's pretty clear what it does. It's also C#, so building up to a less trivial task will be much less horrific than find . -type f -name '*.pgn' -print0 | xargs -0 -n4 -P4 mawk '/Result/ { split($0, a, "-"); res = substr(a[1], length(a[1]), 1); if (res == 1) white++; if (res == 0) black++; if (res == 2) draw++ } END { print white+black+draw, white, black, draw }' | mawk '{games += $1; white += $2; black += $3; draw +…

In a real production environment that command line would be put into a script parametrized with named variables and the embedded awk scripts would be changed to here-docs.

Are you arguing that shell scripts scale to larger applications better than C#?

Re: Command-line tools can be faster than your Hadoop cluster

#134

Post by profquail »

MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

archive.org has Ted's blog post, "Taco Bell Programming": https://web.archive.org/web/20101025124303/http://teddziuba....

Re: Command-line tools can be faster than your Hadoop cluster

#135

Post by dundun »

smegel wrote: post #47

What about if you are processing 100 Petabytes? And you are comparing to a 1000-node Hadoop cluster with each node running 64 cores and 1TB of main memory?

Then you're hardly using commodity hardware anymore. While jobs like that probably actually work on Hadoop, I'd imagine a problem like that might be better suited for specialized systems.

Re: Command-line tools can be faster than your Hadoop cluster

#137

Post by chrishynes »

alrs wrote: post #105
hanoz wrote: post #63

Earlier quoted context omitted.

Your CSV peeking epiphany was in essence a matter of code vs. tools though rather than necessarily CLI vs. GUI. On Windows you might just as well have discovered you could fire up Linqpad and enter File.ReadLines("massive.csv").First() for example.

The example was a multi-gigabyte CSV file. You just sucked the whole thing off the disk into RAM so that you could shave off the first line. If you're unlucky, you started swapping out to disk about halfway through.

Wrong. ReadLines returns an IEnumerable and lets you read line by line without loading the entire file into memory: http://msdn.microsoft.com/en-us/library/dd383503%28v=vs.110%....

Re: Command-line tools can be faster than your Hadoop cluster

#138

Post by linuxhansl »

So don't use Hadoop to crunch data that fits on a memory stick, or that a single disk spindle can read in few seconds.

Why is this first on the HN front-page?

Reminds me of the C++ is better than Java, Go is better than C++, etc, pieces.

Yes, the right tool for the right job. That's what makes a good engineer.

Somebody who thinks there is _no_ valid use case for Hadoop is a fool. (The author did not say that, but many of the comments here seem to imply that view)

Re: Command-line tools can be faster than your Hadoop cluster

#139

Post by JustSomeNobody »

threeseed wrote: post #90
MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

I hate developers who over engineer everything and then when it's time to perform some of that support and extensibility, they leave because maintenance is beneath them.

They put this behemoth together with a thousand moving parts and then walk away from it.

This, too, happens often.

Re: Command-line tools can be faster than your Hadoop cluster

#140

Post by developer1 »

saidajigumi wrote: post #103
threeseed wrote: post #90

Earlier quoted context omitted.

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

> Dreadful for the long term. Here comes a bubble-bursting: I've lead a team that built data processing tools exactly like this, and the performance and ease of manipulating vast amounts of text using classic shell tools is hard to beat. We had no problems with any of: operational supportability, restart recovery, or maintainability. Highly testable, even. No, it's not just cowboy-coded crappy shell scripts and pipel…

The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example, but it's very common for shell scripts to have horrible error handling and checks for important things. The shell is just not suitable for extremely robust programs. I would bet that 80%+ of people who think they're good at shell scripting... aren't.
Post reply on HN