[deleted]
Command-line tools can be faster than your Hadoop cluster
181–190 of 315 posts
Re: Command-line tools can be faster than your Hadoop cluster
#182To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…
Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…
I've implemented this. It isn't too bad up to a certain point. You have to be a bit careful about your filesystem/layout of files, lots of filesystems don't particularly like it when you have a few hundred million files in one directory.
Re: Command-line tools can be faster than your Hadoop cluster
#183Earlier quoted context omitted.
Your CSV peeking epiphany was in essence a matter of code vs. tools though rather than necessarily CLI vs. GUI. On Windows you might just as well have discovered you could fire up Linqpad and enter File.ReadLines("massive.csv").First() for example.
Running a shell in a GUI doesn't make it lose its "I am a CLI" property. That is a CLI.
I disagree. It's a REPL, but a REPL is not always a CLI.
(Frankly, most REPLs are smarter than shells. I go to irb way more than I do bash, these days.)
Re: Command-line tools can be faster than your Hadoop cluster
#184Hi all, original author here. Some have questioned why I would spend the time advocating against the use of Hadoop for such small data processing tasks as that's clearly not when it should be used anyway. Sadly, Big Data (tm) frameworks are often recommended, required, or used more often than they should be. I know to many of us it seems crazy, but it's true. The worst I've seen was Hadoop used for a processing task…
Author of Data Science at the Command Line here. Thanks for the nice blog post and for mentioning my book here. While we're talking about the subject of education, allow me to shamelessly promote a two-day workshop that I'll be giving next month in London: http://datascienceatthecommandline.com/#workshop
Re: Command-line tools can be faster than your Hadoop cluster
#185If bash is the shell (assuming recursive search is required), maybe it would be even faster to just do:
shopt -s globstar
mawk '/Result/ {
game++
split($0, a, "-")
res = substr(a[1], length(a[1]), 1)
if(res == 1)
white++
if(res == 0)
black++
if(res == 2)
draw++
} END {
print game, white, black, draw
}' **/*.pgn
?Re: Command-line tools can be faster than your Hadoop cluster
#186To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…
Alternative, real life scenario: navigate through 6 months of daily MySQL dumps, assorted YAML files and Rails production.log, looking for some cross product between tables, requests and serialised entities, for analysis and/or data recovery (pinpoint or retrieval).
zcat/cut/sed/grep/awk/perl crawled through it in a couple of minutes and required less than half an hour to craft a reliable enough implementation (including looking up relations from foreign keys straight from the SQL dumps).
My colleagues, who still don't get the point of a command line, would still be restoring each dump individually and running SQL requests manually to this day (or more probably declare it "too complex" and mark it as a loss). Side note: I'm torn between leaving this place where nobody seems to understand the point of anything remotely like engineering or keeping this job where I'm obviously being extremely useful to our customers.
Re: Command-line tools can be faster than your Hadoop cluster
#187Bottom line is - you do not need hadoop until you cross 2TB of data to be processed (uncompressed). Modern servers ( bare metal ones, not what AWS sells you ) are REALLY FAST and can crunch massive amounts of data. Just use a proper tools, well optimized code written in C/C++/Go/etc - not all the crappy JAVA framework-in-a-framework^N architecture that abstracts thinking about the CPU speed. Bottom line, the popular…
What makes 2TB the cutoff?
I think the consensus is that as long as your data fits on a single (affordable) machine, 'big data' tools are probably not the best solution.
Re: Command-line tools can be faster than your Hadoop cluster
#188Earlier quoted context omitted.
> Dreadful for the long term. Here comes a bubble-bursting: I've lead a team that built data processing tools exactly like this, and the performance and ease of manipulating vast amounts of text using classic shell tools is hard to beat. We had no problems with any of: operational supportability, restart recovery, or maintainability. Highly testable, even. No, it's not just cowboy-coded crappy shell scripts and pipel…
The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example, but it's very common for shell scripts to have horrible error handling and checks for important things. The shell is just not suitable for extremely robust programs. I would bet that 80%+ of people who think they're good at shell scripting... aren't.
> The shell is just not suitable for extremely robust programs.
Absolute statements like this are usually wrong. This one does not escape the rule. When Linux distros init is mostly bash scripting, there is very little need to further prove that robust systems can be written in bash scripting without the language fighting the developer.
Re: Command-line tools can be faster than your Hadoop cluster
#189Earlier quoted context omitted.
Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…
I look at this article as a criticism of the hadoop being the wrong tool for small data sets. This starts to become a question of data locality, and size. 1.75 GB isn't enough data to justify a hadoop solution. That data size fits easily in memory, and without doubt on a single system. From that point you only need some degree of parallelism to maximize the performance. That being said when its 35TB of data, the answ…
> That data size fits easily in memory, and without doubt on a single system. From that point you only need some degree of parallelism to maximize the performance. That being said when its 35TB of data, the answer starts to change.
Not at all, because data is being streamed. It could just as easily be 35TB and only use a few MB of RAM.
Re: Command-line tools can be faster than your Hadoop cluster
#190Earlier quoted context omitted.
Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…
'pv' shows a progress bar, something like '&& touch $x.success' can help restart recovery. I'd probably pick the shell approach for something I expect to be a one-off, but reconsider each time the task is repeated. I printed http://xkcd.com/1205/ and stuck it to the wall. It's a useful reference when someone seems to be under or overengineering something.
While it drives some point home, the chart eludes the question of robustness (a written script will run twice the same way, whereas human error, especially on routine tasks, may hit one hard) and documentation (writing even a lightly commented script to do yearly maintenance is guaranteed to help your future self remember things one year from now).