Was MongoDB Ever the Right Choice?

Could you elaborate please on the untold pain part? We use Mongo as our primary storage at the moment and I would like to avoid painful issues. Thank you in advance.

On the other note, I keep seeing people recommending Postgres, but to me that is apples vs. oranges. Just because it has json storage doesn't make it a replacement. How much effort and money a company needs to keep a replicated Postres cluster vs. Mongo? In one of my previous jobs we had to hire a consultant to do that. With Mongo pretty much any experience dev could do that.

My personal view is that MongoDB's main advantage is its flexibility. It allows teams to shape and evolve the product as needed. When specific features/parts require new solutions, new solutions can be used (financial transactions, etc). Features that allowed us to leverage what we have instead of researching new tools were:

1. Schemaless - Logic is in the app; Gives visibility; Can be tested; Less migration headaches; Easier to evolve your architecture.

2. Indexed arrays, attributes (name, value pair) - Allows to be more creative (add tagging system to any type of data you have)

3. Aggregation - basic BI could be done from day one, which business can get value sooner.

Of course, everyone's millage varies

The main use case for MongoDB I have seen are custom forms or data that can be nested a variable number of times.

The main problems I have had with MongoDB were that, as of several years ago, it did not integrate well with most (I would argue any in practice) data reporting / displaying third parties.

Also, some of the more advanced queries were not at all intuitive. In fact, I barely remember any of the syntax now. In Mongo's defense, that might have to do with the fact that we atempted some stuff in MongoDB that we would never attempt to do with SQL cursors.

How do you _know_ you never lost data?

You and I must be the only ones on HN using mongo at scale and enjoying it.

We use it as an event database which collects over 100M+ semi-structured records daily with about 200 (and growing) different schemas... It keeps 1.5TB of records in the collection which is achieved using the invaluable capped collection function, and we can index the structured fields very easily. We also pipe the data into elastic for quick kibana querying but just that step requires a lot of index partitioning and mapping to work smoothly.

I also feel that a significant amount of mongo's power lies in the aggregation pipeline which is often ignored and which can replace entire ETLs, sending the compute to the data can be much more efficient in a lot of use cases.

All this runs for us on some relatively cheap (compared to 1.5TB of high i/o RDS) commodity hardware, running 3 replicated nodes. More complex ETL jobs or Hadoop users can pull data from mongo selectively and quickly - spark has a way to partition a query into N threads based on a provided key achieving line rate data pulls even from a single node.

Throw in compression on disk and in-flight and you get something that is really compelling. We've all run out of disk on a database before, it's never fun.

My one gripe is the learning curve for new developers. I do find that once leveraged, I see a lot more data being stored when it's as simple as nesting an object into your main object and saving it, which data teams like.

Depends on your use cases, but really, it's been an invaluable tool so far.

If you need limited document storage capabilities I'd recommend just using S3 alongside a traditional RD like PostgreSQL. You can simply store object IDs in the primary database but the actual document/data in S3. I used this methodology for a cloud platform I built that required the ability to store large 3D models uploaded by users. Metadata was stored in PostgreSQL then the actual data in S3. This also facilitated generating a acquisition URL on the server which could be triggered client-side so that after initial creation, there was almost no primary server overhead (bandwidth or storage) for retrieval.