> "I really don't like if somebody by default compares Hadoop with SQL"
The first comparison was to Scala, and then to SQL. The main comparison was to SQL and a Python script. While it does talk more about SQL than other solutions, I don't see that as a default comparison.
> "For those, the in-memory, GPGPU, streams are much better"
I'll go off on a bit of philosophical tangent here. Yes, GPGPUs can be more effective for the task I'm doing, since I'm actually memory bound. However, GPGPUs require dedicated hardware, while the code I work is effective even on laptops with limited GPUs, including web servers.
Ideologically, I prefer to enable single-person developers, and scientists who do not have much training in hardware and network administration. For that situataion, GPGPUs are not "much better", because so long as the performance is fast enough, it doesn't need to be faster. And 100 ms is "fast enough."
> "This way you don't need to transfer a lot of data"
True. But to point out, I used the traditional approach of developing a file format which can be memory-mapped directly to my internal data structures, and a search algorithm which doesn't need to search the entire data set before loading it. These optimizations aren't synergistic with how I understand how Hadoop works.
> Hadoop's tragedy ... mostly driven by technologically clueless people
The author of the essay agrees with you. Here's the P.P.S.:
"I don’t intend to hate on Hadoop. I use Hadoop regularly for jobs I probably couldn’t easily handle with other tools. ... Hadoop is a fine tool, it makes certain tradeoffs to target certain specific use cases. The only point I’m pushing here is to think carefully rather than just running Hadoop on The Cloud in order to handle your 500mb of Big Data at an Enterprise Scale."
This is why I don't think you actually disagree with the author.
> Think about your case for chemical structures - is there any part of your algorithm that needs to be computed only occasionally but it's a lot of data to process?
Certainly, but there are two other important facets to that. 1) updates occur weekly, a full rebuild on a single core only takes 12 hours, and those 12 hours aren't critical, and 2) the deltas are relatively small and data from previous builds can be reused, so incremental updates should take about 30 minutes - I'll be working on that code in the next couple of weeks.
It's easier to have a cron job trigger a command-line program every week than to set up a Hadoop server.