Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

261–270 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#261

Post by pyre »

sillysaurus3 wrote: post #173
coderdude wrote: post #166

Earlier quoted context omitted.

> Give each process a FIFO to read URLs from. Then you choose which FIFO to add a URL to based on the address so that all URLs with the same address are assigned to the same process. I wrote this in a reply to myself a moment after you posted your comment so I'll just move it here: Regarding the last two issues I mentioned, you could sort the list of URLs by domain and split the list when the new list's length is >=…

Open file on disk. See that it's 404. Delete file. Re-run crawler. You'd turn that into code by doing grep -R 404 . or whatever the actual unique error string is and deleting any file containing the error message. (You'd be careful not to run that recursive delete on any unexpected data.) Really, these problems are pretty easy. It's easy to overthink it.

> grep -R 404

This isn't 1995 anymore. When you hit a 404 error, you no longer get Apache's default 404 page. You really can't count on there being any consistency between 404 pages on different sites.

If wget somehow stored the header response info to disk (e.g. "FILENAME.header-info") you could whip something up to do what you are suggesting though.

Re: Command-line tools can be faster than your Hadoop cluster

#262

Post by MichaelGG »

TallGuyShort wrote: post #245
dundun wrote: post #135

Earlier quoted context omitted.

Then you're hardly using commodity hardware anymore. While jobs like that probably actually work on Hadoop, I'd imagine a problem like that might be better suited for specialized systems.

IME, most installations where Hadoop is "successfully" used it's running on pretty high-end machines. "Commodity hardware" really means standard hardware, not cheap hardware (as opposed to buying proprietary appliances and mainframes).

Or it could be a company with 30M records a month that buys 100 x $200 servers off eBay and is still unable to query their data.

Re: Command-line tools can be faster than your Hadoop cluster

#263

Post by module0000 »

threeseed wrote: post #90
MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

We don't really "come up" with the xargs/wget approach. The approach is already there, waiting to be utilized by someone who understands the tools. The "cool kids" don't like(or are not able) to understand the tools.

The author (I think) is trying to point out that these problems are already solved, decades ago, with existing UNIX tools.

Re: Command-line tools can be faster than your Hadoop cluster

#264

Post by konradb »

rakoo wrote: post #216
eru wrote: post #159

Earlier quoted context omitted.

> [...] I wonder why Clojure wouldn't be — and it could probably be as compact, if not terser. Because Clojure is a goo language, that question depends mostly on the libraries available for Clojure. (Whereas some other languages are worse at gluing, so libraries will only help you so far.)

That's if you want to do everything in Clojure, but if Clojure were to be used as a glue language, it seems to me it has a clear syntax to do it : http://clojuredocs.org/clojure.java.shell/sh Someone even went further to make it more useful: https://github.com/Raynes/conch

I'd never heard of conch, thanks loads for the reference; really useful.

Re: Command-line tools can be faster than your Hadoop cluster

#265

Post by 101914 »

MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

Are these "web pages" all on the same website?

If so, using wget is a poor solution. I have not used wget in over a decade but as I recall it does not do HTTP pipelining; I could be wrong on that - please correct me.

I do recall with certainty that when wget was first written and disseminated in the 1990's, "webmasters" wanted to ban it. httpd's were not as resilient then as they are today, nor was bandwidth and hardware as inexpensive.

HTTP pipelining is a smarter alternative than burdening the remote host with thousands of consecutive or simultaneous connections.

Depending on the remote host's httpd settings, HTTP pipelining usually lets you make 100 or maybe more requests using a single connection. It can be acomplished with only a simple tcpclient like the original nc and the shell.

In any event, the line about a "distributed crawler" is spot on. Never understimate the power of marketing to suspend common sense.

Also, I find that I can often speed my scripts up a little by using exec in shell pipelines, e.g., util1 |exec util2 or exec util1 |exec util2.

There are other, better approaches besides using the builtin exec, but I will leave those for another day.

Re: Command-line tools can be faster than your Hadoop cluster

#266

Post by andyidsinga »

charltones wrote: post #212
threeseed wrote: post #90

Earlier quoted context omitted.

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…

There are command line tools available to help the transition from 'hack' one liner to a more maintainable / supportable solution. For instance drake ( https://github.com/Factual/drake ) a 'Make for data' which does dependency checking would allow for sensible restarts of the pipeline. The O'Reilly Data Science at the Command Line book (linked elsewhere in the comments) has a good deal to say on the subject: turning…

I've been using GNU Parallel for orchestrating running remote scripts/tools on a bunch of machines in my compute and storage cluster. Its now my goto tool for almost any remote ssh task that needs to hit a bunch of machines at once.

An excellent tool ...apparently an improvement on xargs even for local parallel tasks ( see http://unix.stackexchange.com/questions/104778/gnu-parallel-... )

Re: Command-line tools can be faster than your Hadoop cluster

#267

Post by known »

JustSomeNobody wrote: post #237
penguat wrote: post #232

Earlier quoted context omitted.

Because in all too many companies, re-writing from scratch is a no-go, no matter how quickly and sloppily an initial solution was thrown together. I've worked on a prototype => production type project, where the throwaway was never thrown away. (the initial team made some mistakes, chief among them was building one prototype of a whole system, rather than one per major risk)

This is a systemic problem. Engineering is always subordinate to business. This simply should not be the case. We desperately need new business organization models.

https://en.m.wikipedia.org/wiki/Triarchy_%28theory%29

Re: Command-line tools can be faster than your Hadoop cluster

#268

Post by juliangregorian »

MrBuddyCasino wrote: post #61

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…

Having written distributed crawlers, saturating the network connection is quite easy to do and is the main reason for even distributing that type of work in the first place.

Re: Command-line tools can be faster than your Hadoop cluster

#269

Post by virmundi »

rkwasny wrote: post #97

Bottom line is - you do not need hadoop until you cross 2TB of data to be processed (uncompressed). Modern servers ( bare metal ones, not what AWS sells you ) are REALLY FAST and can crunch massive amounts of data. Just use a proper tools, well optimized code written in C/C++/Go/etc - not all the crappy JAVA framework-in-a-framework^N architecture that abstracts thinking about the CPU speed. Bottom line, the popular…

My question is what do you mean by 2TB? At my current client, we have 5 TBs of data sitting (that's relatively recent). Before we had 2-ish. However, we had over 30 applications doing complex fraud calculations on that. "Moving data" (data being read and then worked) is about 40 TB daily. Even with SSD and 256 GB of RAM, a single machine would get overwhelmed on this.

If you're only working one app on less than 1 TB, maybe you don't need something as complex as Hadoop. But given that a cluster is easy to setup (I made a really simple NameNode + Two Data nodes in 45 minutes, going cold), it might not be a bad idea.

I'll take this further and say that some tools for Hadoop that are not from Apache are really nice to work with even in a for non-Hadoop work. For example, I've got to join several 1 GB files together to go from a relational, CSV model into a Document store model. Can I do this with command line tools? Maybe. Cascading makes this really easy. Each file family is a tap. I get tuple joins naturally. I wrote an ArangoDB tap to auto load into ArangoDB. It was fun, testable and easy. All of this runs sans-hadoop on my little MBP.

Fun fact about the Cascading tool set is that I can take my little app from my desktop and plop it onto a Hadoop cluster with little change (taps from local to hadoop). Will I do that in my present example? No. Can I think of places where that's really useful? Yes, daily 35 fraud models' regression tests executed with each build. That's somewhere around 500 full model executions over limited, but meaningful data. All easily done courtesy of a framework that targets Hadoop.

Re: Command-line tools can be faster than your Hadoop cluster

#270

Post by ticviking »

jiffytick wrote: post #224
ajuc wrote: post #217

Earlier quoted context omitted.

The problem - you have file, you want to do one thing for lines matching REGEX and other thing for lines not-matching REGEX. How to do it without iterating the file 2 times? You can do while of course, but it defeats the reason to use shell. I would love to have two-way grep that writes matching lines to stdout and nonmatching to stderr. I wonder if grep maintainers would accept new option for grep "--two-way".

awk can write to stderr.

Which is for errors.
Post reply on HN