Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

101–110 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#101

Post by pjc50 »

kyllo wrote: post #65
mech4bg wrote: post #53

Earlier quoted context omitted.

This shouldn't be a surprise. Tons of development is done on Windows. Most game development, obviously Windows app development, .NET websites, etc. There are command line tools there, but in my 10 years of being a Windows developer, GUI tools were more the norm. There's a time and a place for both. Now developing predominantly under Linux, it amazes me how time consuming and clunky some tasks are on the command line…

Windows is just inherently GUI-centric, if you force developers to use Windows they are naturally going to gravitate toward using GUI based tools because Windows command line is so crippled. Tools like Cygwin and Chocolatey are nice in a pinch but they just don't compare to being in a real UNIX environment.

Windows Powershell is actually quite nice and powerful, while also avoiding some of the shell legacy traps around escaping. It suffers from not being very discoverable and not having a community.

Re: Command-line tools can be faster than your Hadoop cluster

#102

Post by x0x0 »

threeseed wrote: post #90
MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

Sure -- you can level multiple complaints.

What about the failed pages? How about shoving those on a queue and retrying n times with an exponential backoff between. What about the total number of failed pages? What about failed pages by site? etc etc etc

But so what -- the principle is still sound. All I described is still a 100 line python script, written in an afternoon, instead of 3 weeks of working bringing up emr, installing and configuring nutch, figuring out network issues around emr nodes talking to commodity internet, installing a persistent queue, performing remote debugging, building a task dag in either code or (god help you) oozie/xml, and on and on.

Re: Command-line tools can be faster than your Hadoop cluster

#103

Post by saidajigumi »

threeseed wrote: post #90
MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

> Dreadful for the long term.

Here comes a bubble-bursting: I've lead a team that built data processing tools exactly like this, and the performance and ease of manipulating vast amounts of text using classic shell tools is hard to beat. We had no problems with any of: operational supportability, restart recovery, or maintainability. Highly testable, even. No, it's not just cowboy-coded crappy shell scripts and pipelines. Sure, there's a discipline to building pipelined tooling well, just as with any other kind of software. Your problems seem to stem from a lack of disciplined software engineering rather than the tools, or maybe just an environment that encouraged high technical debt.

The kicker? We were using pipeline-based tooling ... running on a Hadoop cluster. Honestly, I'm a bit surprised to see such an apparent mindshare split (judging by some recent HN posts) between performant single-system approaches and techniques used in-cluster. The point that "be sure your data is really, truly big data" is obviously well made, and still bears repetition. Yet the logical follow-on is that these technique are even more applicable to cluster usage. Why would anyone throw away multiple orders of magnitude performance going to a cluster-based approach?

Re: Command-line tools can be faster than your Hadoop cluster

#104

Post by threeseed »

x0x0 wrote: post #102
threeseed wrote: post #90

Earlier quoted context omitted.

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

Sure -- you can level multiple complaints. What about the failed pages? How about shoving those on a queue and retrying n times with an exponential backoff between. What about the total number of failed pages? What about failed pages by site? etc etc etc But so what -- the principle is still sound. All I described is still a 100 line python script, written in an afternoon, instead of 3 weeks of working bringing up em…

Anybody can throw some crap together and make it stick. And it's a perfectly valid solution.

My issue is when there is criticism laid against those solutions which are actually engineered in a way that allows for supportability and extensibility. They are arguably far more important than execution time.

Re: Command-line tools can be faster than your Hadoop cluster

#105

Post by alrs »

hanoz wrote: post #63
danso wrote: post #11

I'm becoming a stronger and stronger advocate of teaching command-line interfaces to even programmers at the novice level...it's easier in many ways to think of how data is being worked on by "filters" and "pipes"...and more importantly, every time you try a step, something happens ...making it much easier to interactively iterate through a process. That it also happens to very fast and powerful (when memory isn't a…

Your CSV peeking epiphany was in essence a matter of code vs. tools though rather than necessarily CLI vs. GUI. On Windows you might just as well have discovered you could fire up Linqpad and enter File.ReadLines("massive.csv").First() for example.

The example was a multi-gigabyte CSV file. You just sucked the whole thing off the disk into RAM so that you could shave off the first line.

If you're unlucky, you started swapping out to disk about halfway through.

Re: Command-line tools can be faster than your Hadoop cluster

#106

Post by AnthonyMouse »

threeseed wrote: post #90
MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

The Unix people have thought of these things. You can easily do them with command line tools.

> Operational Supportability: How do you monitor the operation ?

Downloading files with wget will create files and directories as it proceeds. You can observe and count them to determine progress, or pass a shell script to xargs that writes whatever progress data you like to a file before/after calling wget.

> Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ?

wget has command line options to skip downloading files that already exist. Or you can use tail to skip the number of lines in the input file as there exist complete entries in the destination directory.

> Maintainability: Can we run the same application on our desktop as on our production servers ?

I'm not sure how this is supposed to be an argument against using the standard utilities that are on everybody's machine already.

> Extensibility: Can we extend the platform easily to do X, Y, Z after the crawling ?

Again, what? Extensibility is the wheelhouse of the thing you're complaining about.

Re: Command-line tools can be faster than your Hadoop cluster

#107

Post by threeseed »

saidajigumi wrote: post #103
threeseed wrote: post #90

Earlier quoted context omitted.

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

> Dreadful for the long term. Here comes a bubble-bursting: I've lead a team that built data processing tools exactly like this, and the performance and ease of manipulating vast amounts of text using classic shell tools is hard to beat. We had no problems with any of: operational supportability, restart recovery, or maintainability. Highly testable, even. No, it's not just cowboy-coded crappy shell scripts and pipel…

This "orders of magnitude" less performance going to a Hadoop cluster approach is nonsense.

There are plenty of options for Hadoop that make it dramatically faster than the naive example in the article. Spark ? MR-Redis ? Storm ?

Re: Command-line tools can be faster than your Hadoop cluster

#109

Post by paralelogram »

clarkm wrote: post #88
simpleigh wrote: post #19

Earlier quoted context omitted.

The article addresses this: > The -F for grep indicates that we are only matching on fixed strings and not doing any fancy regex, and can offer a small speedup, which I did not notice in my testing. I guess grep is probably clever enough to choose a faster matching algorithm once it's parsed the pattern and discovered it doesn't contain any regex fun.

In general grep seems smart enough that it would do that, but it hasn't been my experience. Just last week I was searching through a couple hundred gigs small xml files. I found that: $ LC_ALL=C fgrep -r STRING . was much faster than plain grep. This was on a CentOS 5 box, so maybe newer versions of grep are smarter. But then again, if I was on a newer box I'd just install and use ack or ag.

LC_ALL=C makes grep faster because text matching is normally locale-sensitive, for example 'S' ('\x53' in Big5) is not a substring of '兄' ('\xA5\x53' in Big5).

Re: Command-line tools can be faster than your Hadoop cluster

#110

Post by gpapilion »

threeseed wrote: post #90
MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

I look at this article as a criticism of the hadoop being the wrong tool for small data sets.

This starts to become a question of data locality, and size. 1.75 GB isn't enough data to justify a hadoop solution. That data size fits easily in memory, and without doubt on a single system. From that point you only need some degree of parallelism to maximize the performance. That being said when its 35TB of data, the answer starts to change.

The fact that shell commands were used makes for an easy demo that might be hard to support, but if a solution were written using a traditional language with threading or IPC instead of relying on hadoop you should always be faster, since you don't incur the latency costs of the network.

Post reply on HN