NoSQL: The Baby and the Bathwater

This is also a common approach for SQL based databases when low latency is needed for common queries.

> SQL DBs take lower volumes of data, loads of CPU and RAM

That depends on the schema and the queries being run on them. Large amounts of CPU is really only needed if there are either 1000s of queries per second or large joins. RAM is very useful for hash joins with medium sized tables (up to a few millions of records). It has some utility for caching indexes or tables that need lower latency than the IO can provide, but that's similar to most NoSQL that I'm familiar with.

Most RDBMS's also come with ACID support, which has a significant cost (especially when writing). That has little to do with the SQL language, though. Spark tends to run on SQL without paying those costs. (More about this in the OA)

> Even when your project is not in the 100 of milions records this can be significant because you then are able to offer a cheaper product than the competition's

For small databases (For large tables (100 million to 10 billion records), good schema design and well written queries may be needed for SQL to perform, but that's not that different from what you're saying about NoSQL. Still, as you approach the upper end of this range, the compromises in needed with regards to denormalization, slack transaction management, etc may come at the cost of eliminating many of the advantages of traditional RDBMS's (such as consistency enforced by the schema definition through normalization and integrity constraints).

More than 10 billion, and traditional RDBMs start to break down, of course (you may need a cluster, and you may need to use sharding or similar methodologies from NoSQL / Big Data or similar paradigms, even if technically still on a RDBMS).

For small-medium sized databases (Btw, I'm not saying that other kinds of databases don't have their place, especially for various types of unstructured data, document oriented data or situations with extreme volumes or throughput requirements. 20 years ago RDBMS's were certainly overused. But 5-10 years ago, the pendulum had swung a bit in the other direction, imo.

A lot of devs that left college around 2010 seems to have jumped on NoSQL databases less because of their strengths than because of how they enabled the devs to trivially persist object oriented data structures with a line or two of code, and because they'd never learned how to use traditional RDBMS's properly.

This is fine as long as they don't need the kind of data consistency features provided by relational data models, but when consistency is needed, it tends to cause unnecessary problems. (Especially as a system ages, dev teams and application logic evolve.)

The reason I started my previous message with "That's interesting", is that your approach to NoSQL clearly shows you have a mature approach to NoSQL, as opposed to those who consider it a silver bullet that makes all complexity go away.

Anyway, it seems that a good design, suitable both to the problems at hand and the technology chosen/available is a universal in the field. Usually more important than specific tech choices.

I maintain clusters with 50+ machines with 128gb+ of RAM...

In case you aren't: I didn't state my argument at all. Unless you dug through my comment history to find my complaints. This is precisely the kind of nonsense you get in response to even stating that you dislike the language.

> SQL is relational algebra. Sure you can write it some other way.

The same applies to general purpose programming languages, yet they have improved immensely since COBOL.

> Lots of projects tried, none succeeded. Wonder why is that?

Because of inertia, the mixed userbase of SQL, and ORMs and things like LINQ. The fact that most projects avoid direct use of SQL when possible is telling enough on its own.