I posted this further down the thread, but I thought I'd share my thoughts on why I like mongo. Most people don't like mongo because 10gen gives the impression that mongo is better than it actually is, many people feel that mongo is not reliable enough for at-scale applications. They're right; it's not. But that's ok, because: Mongo's really great for rapid prototyping. You don't need to worry about updating the sche…
None of the things you listed are any harder with Postgres and SQLAlchemy. Learning to use MongoDB isn't exactly trivial anyway, so why choose the thing that is known to be broken at all, when it's neither easier nor faster?
I'm an engineer at Tokutek I'm confused by your comment. The beginning acknowledges the fact that MongoDB has a weak storage engine, but your conclusion is that, even with a strong storage engine like ours, there is still a problem. What other problems do you see? Are they something we could work on?
This is going to come off as abit negative but I kinda feel it has to be said. I would first like to say I do love the Fractal tree indexing, very cool and could have alot more intesting usecases outside of databases (I'm thinking logical volume/block storage etc.. I'm always thinking in kernel land..) The problem is that Mongo advertised itself as a database and wasn't one. Once you do that reputation of the product…
I've seen a lot of the rest of their code, and most if it is getting better over time, as they grow they're forced to adopt better habits in order to scale their engineering team. I think you're misunderstanding the type of programmers they are. They didn't use mmap because they are sloppy everywhere, they used mmap because their critical innovation was not in storage. What they really thought was valuable, what they wanted to work on, was the query language and cluster management tools, so they did the simplest thing for storage and moved on (personally I don't understand why they didn't just use BDB, maybe they were afraid of transactions, but I suppose everyone has a little NIH syndrome in their database). Now they're a bit locked in to that code, because after bolting on journaling (that architecture is a brilliant but incredibly dirty hack), the code is a mess and I'm sure nobody wants to touch it. In fact most of the other subsystems have been getting cleaner rewrites, except for the storage layer. I think the only way out is a complete replacement, which is what we did so I feel pretty good about that. So I don't know if I'll convince you, but I've read a lot of their code (especially in the last few weeks, I've been backporting things from 2.4), and that's the feeling I get about their history and vision. Hope it gives you some insight.
If this was a proprietary database we'd call that vendor lock-in and advocate an open source solution. 10gen is a company that earns it's revenue from selling support. They are highly incentivized to lure you in and trap you in a situation that requires a lot of consulting.
Or they could just be interested in adding useful features. PostgreSQL has HSTORE which is a useful but proprietary feature. Cassandra has the ability to have Lists/Maps as data types. Again useful but proprietary. If you are that concerned about database independence then do what everyone else does. Use an ORM, minimise coupling in your domain model and do as much as possible in the application layer.
Cassandra support for lists/maps is open-source. Cassandra is Apache Software Foundation project. Where is it proprietary? Or do you mean something else under "proprietary"?
Postgres update performance is pretty bad. When running a big data migration, it's generally faster to copy the old table to a new temporary table and rename the temp table to the old table than it is to run an update.
This is the case with all RDMBS I've used.
RDBMSes need to check for primary key violations, hence read before write. Random access is slow. The fastest you could do is "no read-before-write, append-only writes, compact later" (Cassandra way).
Mongos (the routing for MongoDB when clustering) has a bunch of drawbacks that make MongoDB worse. One of which is dropping all connections when a master switches. Have you dealt with those problems yet? MongoDB is generally perfect at small scale is what I have perceived. The new database level lock in 2.2 is also annoying (and arbitrary) but it is better than the global lock.
I'm curious, how do other DBMSs handle a master switch/other cluster updates? I'm familiar enough with mongos to know how it works but not what e.g. redis or postgres or mysql does.
The best is not to ever need it by using an architecture with no SPOF (even temporary). Master switch is a huge pain - there are simply too many nasty ways it can fail miserably. I'd stay away from databases needing it, if high availability is the primary concern.
I like your analogy, very fitting. To their defense however they are far from the only ones to do that... I lost about 2h worth of production data with HBase in just the same way - fortunately I didn't want to trust it completely anyway and had my own logs of all transactions on filesystem, but it definitely shattered my trust in that DB (not to mention it was a pain to setup and had no secondary indexes). I use Mong…
doesn't hbase persist in the transaction log every update, using append on hdfs ? or did you have a version of hdfs that didn't have append ?
HDFS didn't have append at the time, not sure how it is now. It did have some filesystem journalling though (if I remember correctly), we just didn't know we should turn it on.
Couldn't you argue that e.g. Postgres and ActiveRecord give you the same rapid prototyping ability but with an easier (and more established) path towards scalability? It is easy to change your schema with migrations at the beginning of a project - just go edit the original ones and nuke your database. And I don't have to worry about properly configuring write-locks, replica sets, or writing map reduce javascript.
Of course you could argue that. But so what? Having an easier path towards scalability is nice, but irrelevant for the vast majority of projects; not every project is going to turn into a startup or a real product or even something you work on for more than a few weekends! The last time you hacked together a blogging engine in Node.js one weekend, were you worried about future scalability, or just playing with new te…
So the argument is that MongoDB is only suitable for weekend hobby projects? Now that's quite a remarkable argument!
Actually postgres can be a bit of a PITA, but so can mongo. At the risk of sounding reckless, unless the app needs to support high CUD throughput I sometimes opt for sqlite. Doesn't get much easier than that and it's read performance is impressive from what I've seen. Even then you can sometimes get away with staying on sqlite for your admin side CRUD and redis for the heavy / concurrent writing from the public facin…
I've been using sqlite more and more as well. Super lightweight, but since it's SQL most ORMs can handle switching to MySQL/postgres really easily if you ever need to make the switch.
Now this is an important point. Once you make your product, what do you need to do to retool a major part of the application? This seems like an excellent approach.
Many people in the Java world use something very simple like hsqldb, then shift to a new database when out of development.