Don't use Hadoop when your data isn't that big

> "Hadoop is not a database!"

Nor does the essay claim that Hadoop is a database.

> "If your problem fits this, Hadoop absolutely rocks"

The essay points out that most systems which do use Hadoop don't actually fit the Hadoop model, and that other 'mature application infrastructures' would be more effective than Hadoop. You misinterpreted it to mean the converse.

It also agrees with you that there are cases where "Hadoop might be a good option". It says "The only benefit to using Hadoop is scaling", and that it might be appropriate for >5TB data sets. Your 1PB example is of course larger than 5TB.

So I don't think you actually disagree with it.

> "Why would you even bother working with toys that put everything into memory and then fail miserably in production"

Thank you for calling my software a "toy" and disdaining my field of research. Do not presume that the needs of your field hold for all others.

Hadoop isn't useful for what I'm interested in, which is to support interactive search of ~2 million chemical structures. This requires a sub-100 ms query time. Hadoop, last I checked, was lousy for soft real-time work like this.

You actually say "Hadoop or any appropriate system that supports your algorithmic model".

My actual solution was the old-fashioned way: a combination of improved algorithms, multithreading, better data locality, and a bit of chip-specific assembly. The result is about 100x faster than the previous widely used tool, and gives me the sub-second search times I want.

Moreover, it scales well. I use the new code as part of the inner loop in a clustering task, what once took a week on a machine cluster is now being done on a single node in an afternoon.

Had I taken your suggestion I would have invested in more hardware, which would have been the wrong solution for my needs. Also, data size in my field doubles every 5-10 years, which is much slower than the rate of computer performance.

For some reason, wrapping my mind around this is like trying to fathom the Grand Canyon or the scale of the universe.

Sorry, I didn't mean to offend you in any way!

First, I really don't like if somebody by default compares Hadoop with SQL and this is a widespread confusion - they are completely different beasts; in fact an extension called Hive gives Hadoop + HTable an SQL-like syntax.

However Hadoop is a parallel, batch-processing platform. Hadoop is slow-responding, you can't even talk about latency because a single task takes a lot of time even to setup/execute. It's completely inappropriate for real-time low-latency computations. For those, the in-memory, GPGPU, streams are much better. However, if you have a large dataset whose loading times onto a single machine may be very long - imagine loading 1TB to memory from a network drive if your computer can handle it - you might be better off by partitioning your data across thousands of smaller nodes (e.g. ARM microservers) and perform computations on each of them independently. This way you don't need to transfer a lot of data, each smaller local dataset is loaded very fast (= you preserver locality), you find balance between being CPU-bound and IO-bound and likely finish your computations much faster than on a single computer with huge memory but limited bus.

Hadoop's tragedy is that it is now an established platform that is supported by large companies which are mostly driven by technologically clueless people, trying to put it everywhere as it is "in vogue" to do so. Google abandoned MapReduce model a few years ago but industry didn't seem to notice.

Think about your case for chemical structures - is there any part of your algorithm that needs to be computed only occasionally but it's a lot of data to process? Why not offload it to Hadoop for preparing digest from these data which you can use in your real-time algorithm? That's actually pretty common usage scenario - Hadoop handles the rough mining, extracting the precious stuff from crude data, assembling it to a form that is refined and can be used by other parts of system that have completely different requirements, for example low-latency interactions.

From my experience SQLite is great when only one process with no parallelism needs to write to the DB and where no other process will frequently overlap reads with writes. However, for CPU-heavy computations that needs multiple cores to compute and write to the DB PostgreSQL has a much better fit for me even for fairly small data-sets. And the ease of use and setup of PostgreSQL is not that much more than SQLite.

This assumes that it's easier to manage 100 servers using Hadoop than, say, using the same 100 EC2 servers with a normal grid scheduler, task queue, or even something like GNU parallel.

It's orders of magnitude easier to start 100 servers and reuse basic Unix skills than it is to setup and manage Hadoop on the same infrastructure.