Live data from Hacker News

12 Months with MongoDB

blog.wordnik.com

61–70 of 75 posts

Re: 12 Months with MongoDB

#61

Alright, so, I fully understand the scalability reasons for using MongoDB, but I need someone to clearly explain to me when NoSQL would be a better solution than SQL from a development standpoint . Because like someone pointed out, Postgres without fsync can be just as fast. What is the advantage of giving up the ability to use SQL and the associated relational algebra that has long been established in that query lan…

Not sure if this is important for you to know, but anyway: I'm one of the creators of Django-nonrel (a fork of Django which adds support for NoSQL DBs to the ORM), I've helped with the development of the MongoDB backend, I've developed a large part of the App Engine backend, and I've worked for over two years with NoSQL solutions.

As you've said, scalability is one valid reason for using a NoSQL solution. Sometimes you also have special needs for multi-datacenter replication or whatever. Sometimes a NoSQL DB can fit your problem better than a SQL DB. However, what does it look like from a development standpoint?

If you have an offline-capable app it might be easier to implement it with CouchDB.

Also, CouchDB's MapReduce views could allow you to run queries that wouldn't be possible with SQL. Note that MongoDB also has MapReduce support, but it rebuilds the whole index instead of only the parts that changed, so on a large DB this might take too long and thus be impractical. Anyway, I guess you're talking about NoSQL solutions in general.

In some cases you actually want to store very flexible schemaless data in your DB. SQL would be a huge pain, here.

It might also be easier to switch from an XML DB to e.g. MongoDB.

Migrations can sometimes also be easier with a schemaless DB. This could be important when experimenting or developing iteratively.

Anyway, most of these are all rather special use-cases and that's probably the important development-side strength of NoSQL: Having solutions for special problems.

Some people here say that working with NoSQL is more natural than with SQL. If you're comparing hand-written SQL code with hand-written code for the low-level MongoDB API then that's indeed true. However, in practice you should better use an ORM because that makes you more effective. When comparing, for example, the Django ORM with MongoEngine you'll see that the APIs look similar. It's just that SQL gives you infinitely more powerful queries and an ORM makes writing even complex queries very easy. A NoSQL DB sometimes forces you to jump through hoops to implement something that would be a two-liner in SQL+ORM. So, from this point of view NoSQL has nothing to offer for a large number of problems. Of course, some projects only use very simple queries which map nicely to NoSQL. In these cases it doesn't make any real difference whether you use SQL or NoSQL, but that still is not a real plus for NoSQL.

One other reason that once spoke for NoSQL: Google App Engine and Amazon SimpleDB were among the first solutions which offered scalable, replicated, managed hosting with an acceptably cheap pay-as-you-go model. Today we have Amazon RDS, MS SQL Azure, and soon App Engine will have SQL hosting, too, and there are several other cheap cloud hosts with SQL support. So, this reason isn't valid, anymore. It might still be part of the hype, though.

Yes, NoSQL is indeed over-hyped. Too many newbies use it for the wrong reasons (esp. on App Engine where they use it because they can save $3/month). Only if you need to scale or if your project has special needs you might find rescue in the NoSQL world.

Finally, since many of us are working on a web startup: As long as it's not clear if you'll become successful and thus have to scale you might be tempted to use SQL and develop faster in the early phase of your project. However, when looking at a complete web project the development/code overhead caused by NoSQL (vs SQL) is often not very large (well, depending on your particular project), so you might want to go with NoSQL right from the start and not have scalability woes later on. Anyway, you need to make an informed decision for your particular problem. Don't just pick NoSQL because it's hyped and also don't just pick SQL because you're wary of hype. :)

BTW, if you want to go with NoSQL and still have more complex queries like with SQL you should join our open-source project django-dbindexer. The goal is to automate denormalization and index generation such that you can use e.g. simple JOINs and aggregates with Django's ORM instead of emulating them with hand-written code. This should be useful to a lot of web startups and make it easier to use NoSQL instead of SQL right from the start and not worry about scalability. See here:

http://www.allbuttonspressed.com/projects/django-dbindexer

http://groups.google.com/group/django-non-relational

Re: 12 Months with MongoDB

#62
post #41

Dumb question: the author talks about not having to use caching because Mongo has built-in caching. Don't most RDBMSes also have built-in caching?

Yes, RDBMDSes have built-in cache, but they are not as fast as memcached. So why the difference? I haven't really checked, but you have to populate and invalidate memcached explicitly, and it doesn't honor database isolation (if you invalidate cache after writing an object, some instances may read stale data after you already updated the db, etc). MongoDB's cache may cut corners in a similar way, I'd love to know.

Re: 12 Months with MongoDB

#63
post #50

Earlier quoted context omitted.

It's a joy to use. We work with objects in our languages. We don't work with rows of data. And I totally get that, I really do, but to me, that's more of an issue of personal preference, and less of an issue of a clear advantage. There's nothing wrong with personal preferences. For instance, I like schemas that aren't easily changed, and a clear separation of logic and data, and I prefer to think of data as rows, not…

You can still think of data as 'rows' with MongoDB - objects belong to a collection, which is analogous to a table - and you can query and select only specific fields. The advantage is that you aren't nearly as restricted in the kinds of data you can build. SQL often forces you to split logically related data across several tables and wrestle with complicated joins, simply because a row can't contain sub-arrays or ha…

simply because a row can't contain sub-arrays or hashes

This is not true at all. At least in PostgreSQL you can have arrays and hashes:

http://www.postgresql.org/docs/current/static/arrays.html

http://www.postgresql.org/docs/current/static/hstore.html

Re: 12 Months with MongoDB

#64

Alright, so, I fully understand the scalability reasons for using MongoDB, but I need someone to clearly explain to me when NoSQL would be a better solution than SQL from a development standpoint . Because like someone pointed out, Postgres without fsync can be just as fast. What is the advantage of giving up the ability to use SQL and the associated relational algebra that has long been established in that query lan…

I am in the very early stages of a project at http://exceptionalasp.net/ and I am using Mongo DB for reporting data and SQL for user accounts etc.

In this case the flexible schema of Mongo DB is a great fit for my data. My records have varying numbers of fields which I don't know in advance, usually include simple hierachies of data and I often need to query all this data together. Many queries that would be difficult in SQL become trivial in Mongo DB.

I use SQL for data relating to user accounts, transactions etc because the SQL model fits better for critical, well defined reliable data IMHO.

Re: 12 Months with MongoDB

#65
post #23

Only had 5 days with MongoDB and I found it a good alternative for persistence of basic data structure in Python, my main concern was something that can $set individual elements of a JSON instead of retrieving the whole doc and modify it.

Not quite sure what you mean, but Atomic Modifications might be what you want: http://www.mongodb.org/display/DOCS/Atomic+Operations

You can update any field in a document. Obviously you'll need the ID of the document.

Re: 12 Months with MongoDB

#66
post #60
post #48

Earlier quoted context omitted.

I do not see how the caching comment here applies, and I think it is telling that this example still includes updating a subtree. I am wondering if you think by "library" I mean "cache layer": I don't. So, either the NoSQL solution you are using is incredibly dumb (and your schema is pretty much "id->blob") or it is internally going to have to do just as many joins against separately stored data objects in order to r…

I'm dealing with this exact problem right now. I'm looking at MongoDB, CouchDB, and Postgres. I agree that Postgres can do this - I've done it before. But I think you're a bit wrong to dismiss document databases so quickly. Firstly, the subtree update problem isn't a huge problem. MongoDB allows dot notation to update items within a document. Yes, it is may well have to do just as much work as a SQL database in the u…

To your "first", I continue to state: that could be handled in a library. There is no reason why this is better handled inside rather than outside of the database. Insisting that this be provided by the database vendor instead of as a layer on top, however, means that you are now taking an entire backend storage implementation (one that is incredibly touchy, I will mind you: I've been using MongoDB in production for the last eight months and I now consider myself an idiot for having wasted time with it) from someone because they provided a convenient syntax.

To your "second": that is not a property of your usage of trees, and starts a new, unrelated discussion. I have nothing against document-oriented databases, and use them often. I feel you are blurring the line between syntax and implementation with your "slow" comment (again: if you are able to concurrently update those schema-less data items you are going to be taking the same hit you would be getting with any other backend for the separate storage and indexing), but will certainly not argue that there are classes of problems where document-oriented databases are really useful. However, trees in particular are not one of their killer features.

Re: 12 Months with MongoDB

#67

Earlier quoted context omitted.

If the multiple outer joins or subqueries are too slow you can always denormalise a bit, you don't have to give up on your SQL database if you don't want to.

I still think it is amazing the the free SQL databases don't have materialized views yet. MySQL's deficiencies aren't inherent to SQL databases. Other databases have faster query parsers and better query planners. It seems that with all the time and money invested into NoSQL solutions, Postgres could be improved to the level of Oracle or DB2.

This comment, which I completely agree with, really depresses me. People really like competing with each other rather than working together to build awesome solutions (and users often even /encourage/ this behavior with a pro-competition bias). The idea that we now have a million "sort of crummy" storage solutions rather than a couple, or even one, really good one--and mostly due to superficial differences in syntax of usage or specification of deployment--makes me want to cry.

Re: 12 Months with MongoDB

#68
post #63
post #50

Earlier quoted context omitted.

You can still think of data as 'rows' with MongoDB - objects belong to a collection, which is analogous to a table - and you can query and select only specific fields. The advantage is that you aren't nearly as restricted in the kinds of data you can build. SQL often forces you to split logically related data across several tables and wrestle with complicated joins, simply because a row can't contain sub-arrays or ha…

simply because a row can't contain sub-arrays or hashes This is not true at all. At least in PostgreSQL you can have arrays and hashes: http://www.postgresql.org/docs/current/static/arrays.html http://www.postgresql.org/docs/current/static/hstore.html

Have you actually looked at or used the array stuff in postgres? It's pretty horrible syntactically and worse, very explicit in it's recommended use.

"Tip: Arrays are not sets; searching for specific array elements can be a sign of database misdesign. Consider using a separate table with a row for each item that would be an array element. This will be easier to search, and is likely to scale better for a large number of elements."

Searching an array is a pretty common task, Mongo does really well in its ability to search into objects in a document.

I had to laugh a little at one of the sample queries:

SELECT f1[1][-2][3] AS e1, f1[1][-1][5] AS e2 FROM (SELECT '[1:1][-2:-1][3:5]={{{1,2,3},{4,5,6}}}'::int[] AS f1) AS ss;

Seriously?

Re: 12 Months with MongoDB

#69
post #68
post #63

Earlier quoted context omitted.

simply because a row can't contain sub-arrays or hashes This is not true at all. At least in PostgreSQL you can have arrays and hashes: http://www.postgresql.org/docs/current/static/arrays.html http://www.postgresql.org/docs/current/static/hstore.html

Have you actually looked at or used the array stuff in postgres? It's pretty horrible syntactically and worse, very explicit in it's recommended use. "Tip: Arrays are not sets; searching for specific array elements can be a sign of database misdesign. Consider using a separate table with a row for each item that would be an array element. This will be easier to search, and is likely to scale better for a large number…

Yes I've looked at them. I use them every day. The example you posted its difficult to parse because it deals with multidimensional arrays and also populating one in the inner query. The simple, and much more common case of one dimension arrays is very straightforward. Can you show me the equivalent in MongoDB?

About the performance/scalability warning; I don't deal with very large arrays, a couple hundred items max, and when using a GIN index over the array field, search queries are screamingly fast.

Re: 12 Months with MongoDB

#70
post #69
post #68

Earlier quoted context omitted.

Have you actually looked at or used the array stuff in postgres? It's pretty horrible syntactically and worse, very explicit in it's recommended use. "Tip: Arrays are not sets; searching for specific array elements can be a sign of database misdesign. Consider using a separate table with a row for each item that would be an array element. This will be easier to search, and is likely to scale better for a large number…

Yes I've looked at them. I use them every day. The example you posted its difficult to parse because it deals with multidimensional arrays and also populating one in the inner query. The simple, and much more common case of one dimension arrays is very straightforward. Can you show me the equivalent in MongoDB? About the performance/scalability warning; I don't deal with very large arrays, a couple hundred items max,…

Sure, check out:

http://www.mongodb.org/display/DOCS/Dot+Notation+%28Reaching...

Post reply on HN