Why SQL is beating NoSQL, and what this means for the future of data

Stonebraker wrote MapReduce: A Huge Step Backwards [1], which basically tells exactly the story you mentioned, the "Hadoop is Codasyl" comparison.

[1] https://homes.cs.washington.edu/~billhowe/mapreduce_a_major_...

I don't think the article could have said it much better.

SQL is super powerful and makes much sense in so many ways. Nearly all apps have a relational structure to them and SQL is a reasonable way to interact with them.

Some of my favorite conversations from the Postgres community 5-6 years back were when they were talking about a time when Postgres was being disrupted. The gray bearded DBAs (Hi Berkus and others) were talking about JSON like it's fad and how it's going to come to pass. They were saying so because they'd heard about this disruption before... there was this new thing XML and these document databases were so much more powerful. They waited a few years and added an XML datatype [1]... And then XML databases came and passed.

At first they scoffed a bit on JSON as the new hip thing, but came around a little in Postgres 9.2, and then in 9.4 [2] we all really got what we wanted.

SQL has never been a particularly elegant language, but it's always been a powerful one. The lone knock against has either been usability or scalability [3]. On the usability side that's a bit of a hard one as well it's not super pretty. But once you understand it's foundations around relational algebra or relational calculus you're so far ahead. On the scalability side there are definitely some options such as Citus, but also core Postgres is massively improving here.

[1] https://www.postgresql.org/docs/9.3/static/datatype-xml.html

[2] https://blog.codeship.com/unleash-the-power-of-storing-json-...

[3] https://www.citusdata.com/blog/2017/02/16/citus61-released/

No tool is good when abused. NoSQL was a knee-jerk reaction and the truth lies somewhere in the middle. Putting everything in relational databases is as bad as putting none of it there. If everything is treated like a tool the world becomes brighter at the cost of having to learn and understand more.

Caveat, this comment isn't directed at you (I agree with your comment), but rather the points around what you are saying.

One thing that helps is if people stop referring to things as SQL / NoSQL as what ends up happening is various things get conflated.

When talking about stores, it's important to be explicit about a few things:

1. Storage model

2. Distribution model

3. Access model

4. Transaction model

5. Maturity and competence of implementation

What happens is people talk about "SQL" as either an NSM or DSM storage model, over either a single node, or possibly more than that in some of the MPP systems, using SQL as an access model, with linearizable transactions, and a mature competent implementation.

NoSQL when most people refer to it can be any combination of those things, as long as the access model isn't SQL.

I work on database engines, and it's important to decouple these things and be explicit about them when discussing various tradeoffs.

You can do SQL the language over a distributed k/v store (not always a great idea) and other non-tabular / relational models and you can distribute relational engines (though scaling linearizable transactions is difficult and doesn't scale for certain use cases due to physics, but that's unrelated to the relational part of it).

Generally people talk about joins not scaling in some normalized form, but then what they do is just materialize the join into whatever they are using to store things in a denormalized model, which has its own drawbacks.

As to the comment above you, SQL vs NoSQL also doesn't have anything to do with the relative maturity of anything. Some of the newer non-relational engines have some operational issues, but that doesn't really have anything to do with their storage model or access method, it just has to due with the competence of the implementation. MongoDB is difficult operationally not because it's not a relational engine, but because it wasn't well designed.

Just like people put SQL over non-tabular stores, you can build non-tabular / relational engines over relational engines (sharding PostgreSQL etc.). In fact major cloud vendors do just that.

ORMs lure devs into thinking you can have a single schema for your app. At launch, this might work. Once you scale, though, ETL is a fact of life. And running a job from you daily stats to the weekly stats, and then to the monthly, quarterly, etc. becomes a requirement.

Which isn't tough, but is often unplanned.