Why I Migrated Away From MongoDB

Be a little careful with this level of disdain.

Sharding may have been available in the 1980's, but what it lead to in some products is quite amazing. Consider Teradata's clustering ability which is sort of like sharding your database but without the disadvantages typically associated with it. Postgres-XC now offers something similar as a semi-fork of PostgreSQL.

Basically what we are talking about here is a two-tier database layer where storage and coordination are separated, and two phase commit is used between these two. Thus the coordination tier can enforce referential integrity between storage nodes if necessary and thus allow write-extensibility.

This isn't something without uses. For high-end, high-write-load databases, serving very large amounts of traffic (think airline reservations), this has been a typical approach for quite some time.

The fundamental problem though is that once you give up on local consistency over a given domain, you cannot have any guarantee of global consistency. The current relational approaches (Postgres-XC and Teradata) both enforce ACID compliance. BASE doesn't offer any consistency guarantee and therefore it is only good for throw-away data.

Yes, as a pre-processing or post-processing layer for an RDBMS ;-). It might also be a useful backplane for some kinds of applications, but most of those are going to have some sort of fixed schema, so you run into a bunch of issues with this.

Another area is for high-write throw-away data. If you are facebook, do you really care if every message gets through or every feed element gets propagated?

Also perhaps content management (which is a write seldom, read often field) might be good, particularly where data can be reconstructed from other sources. Maybe doing the writes first in an RDBMS and pushing out to MongoDB might be a good thing there? You could then do things like referrer tracking and log analysis back in your RDBMS.

Although hybrid databases are way cooler. One idea we are throwing around regarding LedgerSMB development right now is using JSON types in PostgreSQL 9.2 to store extended attributes, so you could have a customer account, and decide you want to store some extra data about this that we don't support right now, and have it stored in JSON.

The basic tradeoff is between flexible input and flexible output. I do think it is sound advice to push the relational database model (this overlaps but is not identical to the relational algebra model) to the breaking point before going elsewhere. If you don't do data validation you can't do ad hoc data transformation later reliably. Strict schemas are always an investment in the future for that reason.

> Polymorphic structures are a PITA to model in relational tables, but they fit very well in schemaless, document-oriented NoSQL systems. This has nothing to do with data validation.

JSON and XML types in PostgreSQL rock. If you can limit the areas where the data is polymorphic, you can push the rest into those types. Many other RDBMS's support XML types as well.

> Rapidly changing schemas are a PITA in RDBMSes, especially with a large cluster of servers. It's usually very easy to alter schemas on-the-fly with NoSQL systems. This has nothing to do with data validation.

This again trades one set of problems for another. You are allowing old schema and new schema to co-exist. How many old schemas does your application have to support? It seems to me this would likely ossify things down the road a bit because if your schema is that rapidly changing, you may never get caught up and so backward compatibility is going to be a bitch. In other words, the less guarantees you can make regarding information stored, the fewer guarantees you can make about data output. That's a big deal as your product matures.

And don't get me started on BASE.....

I do think that the popularity of MySQL, however, owes a lot to it being used by non-engineers for simple web apps, though ;-)

>1. Software gets large / complex to get meaning full work done. I am all about data consistency but at some point it is time to break things up into services and not have a single database.

This is true. Managing complexity is always an important task. I am not sure that NoSQL solves this however. Also the best way to break things up is to loosely couple things, and this requires to some extent that you have ACID compliance. A good RDBMS, like PostgreSQL or Oracle, will provide tools for managing that loose coupling.

>"2. If the software is popular enough everyone is running to use NoSQL (cache is NoSQL)."

Like proverbial lemmings over a cliff....

>"3. Clearly it is not a good storage solution either because for example in an address book nested list greatly simplifies everything. (right tool for the job)"

Funny, I thought nesting was what WITH RECURSIVE was for....

I am not saying there aren't use cases for MongoDB or reasons to switch some applications. For example I can think of a few really cool apps, like maybe a network back-plane for a huge LDAP directory. Also content management might be a good fit. But despite your years of experience, it doesn't sound like you have really looked at how to solve these with good RDBMS's.....