Don't use Hadoop when your data isn't that big

For a lot of people, what the author says is absolutely right, but I think a lot of the comments here suggesting that only a handful of institutions are solving Hadoop-scale problems is simply inaccurate.

Yes, there are companies that are trying to appear more attractive by using Hadoop, but there are plenty of cases where Hadoop is replacing ad-hoc file storage on multiple machines.

It's primary use is as a large scale filesystem, so if you are running up against problems storing and analyzing data on a single box, and you feel that the amount of data you have will continue to accelerate, it is a good option for file storage. It doesn't replace your database, it complements it, and there's work being done to allow large-scale databases on top of Hadoop, although the existing ones aren't mature yet. But there's a lot of institutions that are taking on problems that a single-box setup cannot handle.

And MapReduce isn't a bad programming model, but it should be thought of as the assembly language of Hadoop. If you are solving a particular problem on Hadoop, writing a DSL for it is the way to go, or see if one of the existing DSLs fits your needs (HIVE, Pig, etc).

This goes back to the old adage "the right tool for the job".

As @davidmr points out, there are jobs on smaller data sets that can still benefit from the distributed nature of HPC.

That said, my own experience with startups echoes much more what OP writes - python scripts and csv processing has saved me days of headaches in resource constrained environments. I was able to quickly produce analysis and make crucial scaling decisions using data that would have taken days of engineering resources to produce. I happened to have the right tool for that job handy, and it worked out great.

You really have to think things through before restricting yourself to any specific direction.

We've had a lot of success with EMR as well - we have an hourly Pig job that produces data for our analytics database. It's not a particularly complex script, but our traffic volume is unpredictable so it's reassuring to know that we can add resources to a slow job and have it finish faster.

The downside of EMR is that it can be fairly expensive once you start needing the beefy machines. We're lucky that we can afford to have our analytics delayed an hour or two and can thus run on Spot instances (except for the Master node). When we move to a streaming architecture I'm not sure EMR will still be competitive, since we won't be able to have those machines go away on us.

Edit: clarity.

A study of jobs submitted to the Yahoo! cluster showed that the median job involved 12GB of data.

There's really nothing wrong with that at all, because breaking on 64MB blocks, that 12GB can be processed in parallel, which means turning an answer around really quick on that 12GB, say 30 seconds or so. Usually the work can be scheduled on machines that already have the necessary input, so the network cost is low.

Now, it might not be worth it for one hacker to build a Hadoop cluster to do that one job, but if you have a departmental-wide or company-wide cluster you can just submit your jobs, get quick answers, and let somebody else sysadmin.

Sure the M/R model is limited, but it's a powerful model that is simple to program. You can write unit tests for Mappers and Reducers that don't involve initializing Hadoop at all, and THAT speeds up development.

Yes, it is easy to translate SQL jobs to M/R, but M/R can do things that SQL can't do. For instance, an arbitrary CPU or internet intensive job can be easily embedded in the map or in the reduce, so you can do parameter scans over ray tracing or crack codes or whatever.

I built my own Map/Reduce framework optimized for SMP machines and ultimately had my 'shuffle' implementation break with increasing input size. At that point I switched to Hadoop because I didn't plan to have time to deal with scalability problems.

https://github.com/paulhoule/infovore/wiki

With cloud provisioning, you can run a Hadoop cluster for as little as 7.5 cents, so it's a very sane answer for how to get weekly batch jobs done.

Great points in this article - though using something like cascalog makes hadoop suck a lot less - e.g composable more complex queries closer in power to sql. Wouldn't be quite as crazy to use on smaller datasets be it just for fun or to prove you are ready if/when your dataset grows large enough.