How NoSQL forced the evolution of a scalable relational database

(1) Wide table with 100K analytical features keyed against a customer. Now if you need to combine a few dozen of these features for predictive purposes then how do you do that in SQL ? You can't. Since every SQL database has a tiny column limit. In Cassandra that is a trivial O(1) fetch.

(2) Time series data can be many orders of magnitude faster in JSON document stores where you can have embedded/nested records.

(3) Star schema with a fact table and thousands of dimensions. In a SQL database that is thousand joins. In a JSON document store those dimensions can be embedded as well as made available as relations and again that is an O(1) fetch.

We rarely use anything else, but I can see the value of entity if you’re changing your DB a lot. We typically don’t do that though, and to be honest, working with changes with linq to sql, typically goes smoother than entity migrations.

Disclaimer: it’s entirely possible that we simply suck at entity.

Do you mean Cassandra (BigTable), MongoDB (Document), Riak (Key/Value), Redis (Mix), Kafka (Log Structured) ? There are dozens of fundamentally different systems many of which are closer to an RDBMS than their NoSQL peers. And many of them have rigid schemas so it definitely isn't that either.

I chose Mongo for a project recently. I have many years experience with SQL databases, and I think SQL will become more of a niche in the future.

Here are the reasons:

Everybody uses an ORM with a SQL database. You can pretend they don't and everyone is writing raw sql, but they aren't. This is basically a big, complicated, and slow piece of software that tries to make a SQL database into something else. A NoSQL database is like a native version of that. It starts off with you being able to define collections and schemas in code and so on.

SQL is a shit way to query a database. I used to write massive queries that took an hour or so to write just so that the non-programming people could put them into analytics software that didn't support any other methods of input. While I was writing these queries, I was just thinking to myself "I can write this in code in a few minutes instead of an hour or more". The fact is that programming languages are much better at querying databases than SQL.

Joins are slow and complicated, and ORMs are slow and complicated, so you can never actually use any of these things in a high-traffic environment. As you say, not many people will reach these levels of traffic, but it is still a factor.

Most of the comments about "I moved from mongo to postgres" lately are first-time programmers who didn't know any of the basic concepts of databases. They then discovered patterns that the SQL database forced on them and declared that mongo sucks when in reality, they just didn't know what they were doing.

>One of the problems of many stacks is that the frameworks wrap general purpose languages over SQL, which, is not really a good idea, SQL is a vastly more capable language for dealing with relational data and layers built over the top often dumb down the database.

This is not true at all. Programming languages are vastly superior at querying a database. Just go write a complex query and compare it to the one in linq or whatever you use.

>Microsoft at one stage had Linq to SQL which was quite good..... but they killed it :)

They have entity framework, which uses linq and is the same thing. It is very slow though.

It would be more honest to say that the weird js design, its lack of features and ridiculously small stdlib made it harder to do anything complicated at the time (no ES6 or webpack, remember ?).

Hence, not good tooling existed for relational databases: no decent abstraction layer, migration lib or ORM. Lost of languages had a 2 liner to pop an sqlite db without needing to install anything, but JS didn't even have that.

So here you are, with thousands of front end devs that were just learning to declare their variable correctly, coming to the server. They had the choice between a technology that needed a lot of boiler plate to get anything done, and provided a painful experience with many early errors.

Or something that just seemed to work out of the box. You just dump your thoughts in it. You don't need to think. No error back. It just says "ok".

Seemed is, however, important here. The first years of mongo, it failed miserably, but you only saw it down the road of the projects using it. And of course the inexperience of the css wizards using it as a "dump-my-data" store prove to be a terrible curse.

At the time, if you said any of those things, people would have called you an old grumpy looser that didn't want to live in the future.

But they learned that, just like "var" was important to avoid bugs instead of using your variable whenever you felt like it, deciding of a formal schema was, even for document data store, an important thing to do.

Fast forward, the JS community has now matured and include modern constructs, a rich ecosystem and people with way more experience. Mongo has changed its design and has now sweet spots use cases.

But we still don't have good tooling for relational DB in JS.

And it's still important to have an explicit document stating which schema you use for your document store. Schema less means you can do whatever you want. It doesn't mean, however, that you should.