MongoDB queries don’t always return all matching documents • Hacker News Board

Re: MongoDB queries don’t always return all matching documents

#291

Post by mgkimsal » Wed, Jun 08, 2016, 10:57 AM UTC

Lazare wrote: post #124

spriggan3 wrote: post #76

Earlier quoted context omitted.

> Why on earth would you use a database that makes so little in the way of guarantees about what results you get from it? Because some people can't stand having to work with SQL,migrations,schema and constraints, it's as simple as that ( That's not my opinion,that's just the rational behind MongoDB). Even if you use Postgres with the Json column type, you still need to write SQL queries and schemas. In the context of…

> Because some people can't stand having to work with SQL,migrations,schema and constraints The thing is, if you actually try and write an app using MongoDB, you will rapidly find that you: 1) Have migrations (except they're going to be some scary ad hoc nodejs script that loop through your document store and modify fields on the fly). 2) Have schemas (except they'll be implicit and undocumented) 3) Constraints (exce…

#2 - it's basically "schema on read" vs "schema on write".

You always have a schema; where and how it's defined is the only question.

Re: MongoDB queries don’t always return all matching documents

#292

Post by 1879325238 » Wed, Jun 08, 2016, 11:03 AM UTC

orf wrote: post #261

1879325237 wrote: post #250

Earlier quoted context omitted.

> I've written a ton of SQL migrations this year, none of them are scary or even really count as "big bunch of sql". If this describes your code, you should probably stop and rethink what you're doing, because that's bad practice. The difference between, eg, merging your "first_name" and "last_name" fields into a single "name" field with Mongo or a SQL DB is basically that the SQL DB has better tooling so it'll be cl…

Tl:dr - you used a crappy ORM and now think all ORM libraries sick. So... use MongoDB... which is always accesses through an ORM... All your comments come across as confused mate. Enforcing arbitrary constraints belongs in the business logic of your app, but dealing with consistency issues does not. Postgres handles row constraints just fine, you can enforce any arbitrary logic (i.e colX != colY and its not the first…

Mongodb isn't always accessed through an ORM. I have no idea what you are talking about.

> Enforcing arbitrary constraints belongs in the business logic of your app

Oh so you have some constraints enforced in your database, like the length of a strength can't be greater than 5, yet others you consider arbitrary just because they are not in SQLs list of constraints? How on earth did you define arbitrary otherwise? Anyway, that is the entire point, SQL's constraints don't encapsulate all of the constraints, so you STILL HAVE TO MANAGE MANY CONSTRAINTS IN THE CODE. I hope you got it that time....

> but dealing with consistency issues does not

Yes that is why I said you have to deal with consistency issues in Mongo. I then referred to consistency in another context, which was confusing, I understand. I was referring to consistency on a higher level, in terms of an entry called "Can't be deleted", with a Deleted flag set to true. Which is not related to the data consistency I was referring to earlier in the paragraph. That does not invalidate any of my arguments, you just wish it did.

Re: MongoDB queries don’t always return all matching documents

#293

Post by chris_wot » Wed, Jun 08, 2016, 11:05 AM UTC

fortytw2 wrote: post #273

dijit wrote: post #239

Earlier quoted context omitted.

Well, that's not entirely true. if you're on a mac you can download postgresql.app[0] which produces a small icon in the top right status bar. You don't have to install users or permissions or anything it's super easy to set up. Getting it on prod can come later but for the first five minutes it works. (granted, this neglects contrib extensions like hstore) [0] http://postgresapp.com/

That sets up postgres. It doesn't let you get started doing CRUD operations inside your postgres though - which is where MongoDB shines, "just store my data, fuck it"

Oh, it stores data, just don't try to get it out again...

Re: MongoDB queries don’t always return all matching documents

#294

Post by [deleted] » Wed, Jun 08, 2016, 11:05 AM UTC

fortytw2 wrote: post #273

dijit wrote: post #239

Earlier quoted context omitted.

Well, that's not entirely true. if you're on a mac you can download postgresql.app[0] which produces a small icon in the top right status bar. You don't have to install users or permissions or anything it's super easy to set up. Getting it on prod can come later but for the first five minutes it works. (granted, this neglects contrib extensions like hstore) [0] http://postgresapp.com/

That sets up postgres. It doesn't let you get started doing CRUD operations inside your postgres though - which is where MongoDB shines, "just store my data, fuck it"

[deleted]

Re: MongoDB queries don’t always return all matching documents

#295

Post by 1879325238 » Wed, Jun 08, 2016, 11:12 AM UTC

DoofusOfDeath wrote: post #284

1879325237 wrote: post #250

Earlier quoted context omitted.

> I've written a ton of SQL migrations this year, none of them are scary or even really count as "big bunch of sql". If this describes your code, you should probably stop and rethink what you're doing, because that's bad practice. The difference between, eg, merging your "first_name" and "last_name" fields into a single "name" field with Mongo or a SQL DB is basically that the SQL DB has better tooling so it'll be cl…

> No the difference is one has to be written in a shitty language (SQL), whereas the other can be written in an actual programming language. SQL is a DSL. It has pros and cons for data manipulation. For many combinations of (particular programmer) + (particular thing to get done) + (set of considerations), SQL turns out to be a better tool than a more general-purpose programming language. YMMV, obviously. > Err no it…

> SQL is a DSL. It has pros and cons for data manipulation. For many combinations of (particular programmer) + (particular thing to get done) + (set of considerations), SQL turns out to be a better tool than a more general-purpose programming language. YMMV, obviously.

You can argue how great SQL is as a language all you like. Hands up everyone in here who likes writing raw SQL.. lets point out how unrepresentative this audience is.

> It sounds like you're making a decent argument against manually modifying a database schema at the SQL level, behind the back of some ORM's which can't gracefully handle such surprises. But I don't think anyone is disagreeing with you about that.

It still means you have literally two contexts in which to handle migrations. Two completely different ways. That doesn't sound more complex to you?

> Many programmers can handle that complexity.

Me too.. I know everything there is to know about SQL, have been programming and using SQL databases for decades. You were arguing that it was easy then something like this:

class House { Array[] rooms; }

class Room { House house; }

It isn't simpler.

Re: MongoDB queries don’t always return all matching documents

#296

Post by dkersten » Wed, Jun 08, 2016, 11:18 AM UTC

1879325237 wrote: post #250

Lazare wrote: post #223

Earlier quoted context omitted.

> Unless you have never written migrations in SQL before you would know that they are even scary and a big bunch of sql I've written a ton of SQL migrations this year, none of them are scary or even really count as "big bunch of sql". If this describes your code, you should probably stop and rethink what you're doing, because that's bad practice. The difference between, eg, merging your "first_name" and "last_name" f…

> I've written a ton of SQL migrations this year, none of them are scary or even really count as "big bunch of sql". If this describes your code, you should probably stop and rethink what you're doing, because that's bad practice. The difference between, eg, merging your "first_name" and "last_name" fields into a single "name" field with Mongo or a SQL DB is basically that the SQL DB has better tooling so it'll be cl…

written in a shitty language (SQL)

I disagree with this. After doing a bunch of work with SQLAlchemy and Hibernate over the years, I've developed a major dislike to ORM's as a replacement for SQL and would much rather write my queries in plain SQL. My currently favourite SQL library is HugSQL[1], which lets you write your queries in SQL in .sql files and HugSQL will load them and provide functions which you can call to invoke the query.

I find SQL quite solid for querying and updating data. Its what it was designed to do. Much more solid that the query API's I've seen in ORM's.

However, more importantly, when performance matters, it is very difficult to properly understand performance characteristics of complex ORM queries (I often find myself making the ORM generate raw SQL, which I then manually review to determine what its doing (eg what indexes it uses, etc)) and I find it much easier to optimise SQL than ORM calls.

My point isn't to badmouth ORMs. If you like them and they solve your problems, that is awesome. My point is that calling SQL a "shitty language" is purely subjective and your opinion and other people have different opinions. SQL is not an objectively shitty language. I think its a pretty good data query and manipulation language. Not perfect, but definitely not shitty.

And no, I'm not a DBA. I'm a programmer who hates having to mess around with databases. It is for this reason that I want things to just stay out of my way so I can get my work done with minimal touching of the database. I've recently found myself forced to do things I never wanted to do (1. change data capture on a MySQL database; 2. query and table optimisation to make a database perform and scale better) and for #2, my life would have been a lot easier if I could have worked with raw SQL.

[1] http://www.hugsql.org/

Re: MongoDB queries don’t always return all matching documents

#297

Post by chris_wot » Wed, Jun 08, 2016, 11:19 AM UTC

1879325238 wrote: post #292

orf wrote: post #261

Earlier quoted context omitted.

Tl:dr - you used a crappy ORM and now think all ORM libraries sick. So... use MongoDB... which is always accesses through an ORM... All your comments come across as confused mate. Enforcing arbitrary constraints belongs in the business logic of your app, but dealing with consistency issues does not. Postgres handles row constraints just fine, you can enforce any arbitrary logic (i.e colX != colY and its not the first…

Mongodb isn't always accessed through an ORM. I have no idea what you are talking about. > Enforcing arbitrary constraints belongs in the business logic of your app Oh so you have some constraints enforced in your database, like the length of a strength can't be greater than 5, yet others you consider arbitrary just because they are not in SQLs list of constraints? How on earth did you define arbitrary otherwise? Any…

What do you mean by consistency? Something tells me you aren't using it in the way most of us understand it...

Re: MongoDB queries don’t always return all matching documents

#298

Post by snaky » Wed, Jun 08, 2016, 11:20 AM UTC

1879325235 wrote: post #204

> 1) Have migrations (except they're going to be some scary ad hoc nodejs script that loop through your document store and modify fields on the fly). Unless you have never written migrations in SQL before you would know that they are even scary and a big bunch of sql. What is even worse is that, despite nobody here acknowledging it, must SQL databases are used with an ORM. Now in your migration script you have to hac…

> SQL constraints are very limited

Oh really? What exactly cannot be expressed in PostgreSQL check constraint, given the fact you can write it in any programming language you prefer?

Re: MongoDB queries don’t always return all matching documents

#299

Post by gaius » Wed, Jun 08, 2016, 11:21 AM UTC

joncrocks wrote: post #279

gaius wrote: post #257

Earlier quoted context omitted.

Oh, that doesn't happen much, because serious organizations have dozens of apps connected to a DB and no-one would be crazy enough to big-bang change them all at once to use a different schema. Adding columns and tables tho' is easy, populating new tables from existing, replacing tables with views, no problem.

I'd argue the inverse. Serious organisations shouldn't have lots of different apps reading from each other's databases, for the reason you suggest above. It makes changes difficult and the application landscape fragile. It also encourages not having well-documented interdependencies between your apps (and consequently no way of knowing what the impact of any given change to an app will have). It's not uncommon howeve…

They are not "each others" databases tho'. It is "the database", deliberately chosen as the integration point. The alternative is a horrific tangle of replication, or a vast undiscoverable landscape of tiny APIs.

Re: MongoDB queries don’t always return all matching documents

#300

Post by varunom » Wed, Jun 08, 2016, 11:24 AM UTC

collyw wrote: post #246

catnaroek wrote: post #131

Earlier quoted context omitted.

Exactly. While the process of designing the structure of your data can make you feel like “you're not getting real work done”, in the long run, it actually prevents headaches caused by inconsistent data. Data always has a structure, it's just that some people are too lazy or mentally feeble to figure out what it is.

For me that is the most important aspect of starting / designing an application. If the data model is accurate, then the code falls into place easily. If its not quite right, more and more code ends up in the application trying to make up for the poor data model.

I second this.

My first task in any project is to design the whole data model based on current requirements and while designing it I think of the interfaces and how would they read and write data (to refine requirements). Writing views and actions/APIs on top of well-formed data model then becomes a breeze.