Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line tools can be faster than your Hadoop cluster

aadrake.com

Post reply on HN
Print view View original

51–60 of 315 posts

Re: Command-line tools can be faster than your Hadoop cluster

#51

Post by ggreer »

kylek wrote: post #36

I feel ag (silver surfer, a grep-ish alternative) should be mentioned (even though he dropped it in his final awk/mawk commands) as it tends to be much faster than grep, and considering he cites performance throughout.

GitHub link for those who don't know about it: https://github.com/ggreer/the_silver_searcher/

I built ag for searching code. It can be (ab)used for other stuff, but the defaults are optimized for a developer searching a codebase. Also, when writing ag, I don't go out of my way to make sure behavior is correct on all platforms in all corner cases. Grep, on the other hand, has been around for decades. It probably handles cases I've never even thought of.

Re: Command-line tools can be faster than your Hadoop cluster

#52

Post by dice »

zobzu wrote: post #43

Heres a probably unpopular opinion.... Pipes make things a bit slow. A native pipeless program would be a good bit faster - incl. an acid db. Note that doing this in python and expecting it to beat grep wont work... The other thing is that hadoop - and some others are slow on big data (peta, or more) vs own tools. Theyre necessary/used because of massive clustering (10x the hardware deployed easily beats making ur ow…

In the final solution at the end of the article there are only two pipes:

1. A pipe to feed the file names into xargs for starting up parallel `mawk` processes.

2. A pipe to a final `mawk` process which aggregates the data from the parallel processes.

There's still some performance that could be gained by using a single processes with threads and shared memory, but this is pretty good for something that can be whipped together quickly.

Re: Command-line tools can be faster than your Hadoop cluster

#53

Post by mech4bg »

Swizec wrote: post #41

Maybe I come from a weird world, or even a weird generation. But when I was in high school, Linux fanboyism was at its peak and just like people get all wound up on bands and such, us geeks got wound up on open-source and linux and fck Micro$oft etc. etc. This was early-ish 2000's. As a result. Every serious programmer I know, especially those who are about my age, lives their life in the CLI. It always comes a surpr…

This shouldn't be a surprise. Tons of development is done on Windows. Most game development, obviously Windows app development, .NET websites, etc.

There are command line tools there, but in my 10 years of being a Windows developer, GUI tools were more the norm.

There's a time and a place for both. Now developing predominantly under Linux, it amazes me how time consuming and clunky some tasks are on the command line compared to using a GUI (e.g. debugging, Visual Studio is just a fantastic IDE), but also how much faster and easier other tasks are with a CLI.

Re: Command-line tools can be faster than your Hadoop cluster

#54

Post by prodigal_erik »

Swizec wrote: post #41

Maybe I come from a weird world, or even a weird generation. But when I was in high school, Linux fanboyism was at its peak and just like people get all wound up on bands and such, us geeks got wound up on open-source and linux and fck Micro$oft etc. etc. This was early-ish 2000's. As a result. Every serious programmer I know, especially those who are about my age, lives their life in the CLI. It always comes a surpr…

People are always surprised when I mention that the Microsoft devs I worked with had free access to the highest tiers of Visual Studio, yet what they actually worked in was vim and the internal fork of make. I don't know whether that's still true; it's been a decade now.

Re: Command-line tools can be faster than your Hadoop cluster

#56

Post by KaiserPro »

radoslawc wrote: post #24
KaiserPro wrote: post #14

Earlier quoted context omitted.

Depends on the dick, depends on the storage. HDFS is a psudeo block interface. If you have a real filesystem like lustre, or GPFS, not only do you have the abilty to use other tools, you can use that storage for other things. In the case of GPFS, you have configurable redundancy. Sadly with lustre, you need decent hardware, otherwise you're going to loose data. In all these things, paying bottom dollar for hardware,…

> Depends on the dick not really, sorry I had to back to the topic, HDFS is really somewhat waste of disk space, especially when used for something like munching logs > At scales of 1pb+ (which is about 1/2 a rack now) its much much cheaper to use off the shelf parts with 24/7 support than "softwareing" your way out. depends, if you need monthly reports from logs, as long as you don't loose storage completely, then u…

Ahem

Disk....

Re: Command-line tools can be faster than your Hadoop cluster

#57

Post by ajdecon »

kunstmord wrote: post #22
volker48 wrote: post #15

Earlier quoted context omitted.

If the data can fit on a thumb drive it's not big data.

I think I read this somewhere here a few months ago (paraphrasing, obviously): "When the indices for your DB don't fit into a single machines RAM, then you're dealing with Big Data, not before."

And following up: Your laptop does not count as a "single machine" for purposes of RAM size. If you can fit the index of your DB in memory on anything you can get through EC2, it's still not Big Data.

Re: Command-line tools can be faster than your Hadoop cluster

#58

Post by ggreer »

smegel wrote: post #47

What about if you are processing 100 Petabytes? And you are comparing to a 1000-node Hadoop cluster with each node running 64 cores and 1TB of main memory?

Right tool for the right job. 100 petabytes is 50,000,000 times larger than the data in the post. It's the difference between touching something within reach and flying around the world.[1]

1. Earth is 40 megameters in circumference. 40Mm / 50M = 0.8m

Re: Command-line tools can be faster than your Hadoop cluster

#59

Post by walshemj »

volker48 wrote: post #15
EpicEng wrote: post #10

Earlier quoted context omitted.

Exactly this just happened where I work. The CIO was recommending Hadoop on AWS for our image processing/analysis jobs. We process a single set of images at a time which come in around ~1.5GB. The output data size is about 1.2GB. Not a good candidate for Hadoop but, you know... "big data", right?

If the data can fit on a thumb drive it's not big data.

if a single file of the data can fit on the single biggest disk commonly available it not big data

Re: Command-line tools can be faster than your Hadoop cluster

#60

Post by theVirginian »

crcsmnky wrote: post #7

Perhaps I'm missing something. It appears that the author is recommending against using Hadoop (and related tools) for processing 3.5GB of data. Who in the world thought that would be a good idea to begin with? The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they…

"Although Tom was doing the project for fun, often people use Hadoop and other so-called Big Data (tm) tools for real-world processing and analysis jobs that can be done faster with simpler tools and different techniques."

I think the point the author is making is that although they knew from the start that Hadoop wasn't necessary for the job, many people probably don't.

Post reply on HN