Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB queries don’t always return all matching documents

engineering.meteor.com

Post reply on HN
Print view View original

411–419 of 419 posts

Re: MongoDB queries don’t always return all matching documents

#411

Post by badminton1 »

btown wrote: post #152
badminton1 wrote: post #150

Earlier quoted context omitted.

Using a document oriented data store as a log aggregator? Please lord, take him in his sleep.

It does work for a certain volume of data. You can index fields you're interested in, even do so after the fact, and it's like any other database in that case. And sometimes you have small apps that do need complete historical log data, so Kafka et al just introduce unnecessary complexity since you'd need to aggregate into a key value store anyways. But if you do this, god forbid you go beyond where indices can fit i…

Try some Lucene based software and tell me how it goes.

Re: MongoDB queries don’t always return all matching documents

#412

Post by fiatjaf »

mikekchar wrote: post #409
fiatjaf wrote: post #372

Earlier quoted context omitted.

one of the things I want to try (but have only done some preliminary trials with) is to keep the concept of the write only ledger, but to replicate the db into several views of the data (some with severely restricted content). How is that different from the current concept of CouchDB views? You meant to replicate the DB to various different places and use the same CouchDB views from there? Or you meant something like…

Yes, the latter. One of the main problems I've seen is that indexing things that you will never query is both expensive in time and space. Also it's amazing how many views tend to have exactly the same data, only sorted differently. And the reason to sort it differently is because you only want to work on a subset of the data, but you can only restrict the query in a contiguous section of keys. An example of this mig…

I like the concept of views. I think it is the best part of CouchDB. Saving data and later defining views (that should be very fast). Once I wrote an app that stored massively enormous documents with lots of data, the views were used to turn that data into queriable information later.

I like what you suggested very much, because what CouchDB can currently do with views is very limited to what a powerful views implementation would, and the one you've suggested is a good suggestion of how to do it better.

Thank you. I'll give this a lot of thinking when I restart writing my https://github.com/fiatjaf/summadb/

Re: MongoDB queries don’t always return all matching documents

#413

Post by nathan_long »

xapata wrote: post #160
red_hare wrote: post #149

Earlier quoted context omitted.

This. My company used mongo for years before we got our shit together. Schmemas were always implicit (until we got our shit together and started defining and enforcing them with Python Schematics). Migrations were crazy scripts you run in prod or hacks you stick into your code to "transition". And yes, surprise constrains left and right causing awful anti-patterns. One-character key names to save disk. Hashed values…

> used mongo for years before we got our shit together. That's actually a legit use case. Use MongoDB while you get your shit together. I use global variables while I'm noodling around in code. Eventually I refactor.

My concern would be whether Mongo will cause me to lose data.... "To recap: MongoDB is neither AP nor CP. The defaults can cause significant loss of acknowledged writes. The strongest consistency offered has bugs which cause false acknowledgements, and even if they're fixed, doesn't prevent false failures." https://aphyr.com/posts/284-call-me-maybe-mongodb

...or get my data corrupted: "When MongoDB is all you have, it’s a cache with no backing store behind it. It will become inconsistent. Not eventually consistent — just plain, flat-out inconsistent, for all time. At that point, you have no options. Not even a nuclear one. You have no way to regenerate the data in a consistent state." http://www.sarahmei.com/blog/2013/11/11/why-you-should-never...

When you refactor or rewrite your code, you have the old code in version control, can write tests to confirm that it still works as expected, and there's no inherent time pressure.

If you pick an unreliable database and your data has been or is being lost and/or corrupted, it's more like a "try to stop the bleeding before the patient dies" situation.

That's not the time I want to be considering changing databases.

Re: MongoDB queries don’t always return all matching documents

#414

Post by nathan_long »

Lazare wrote: post #225
nulagrithom wrote: post #220

Earlier quoted context omitted.

This list makes me want to cry a little. It rings too true. > Have migrations (except they're going to be some scary ad hoc nodejs script that loop through your document store and modify fields on the fly). I literally just spent the better part of tonight AND yesterday evening dealing with one of these scripts. I had pulled down the production table to locally test the script (gross), but when I later ran it in the…

Oh god, I'm getting flashbacks. Because you can't just test it on one document and see if it works; you have no guarantee that all the documents will be identical. And if the migration script crashes halfway through... oh man.

By contrast, with Postgres you'd have an explicit schema which all records obey, and even migrations can be done in transactions.

Re: MongoDB queries don’t always return all matching documents

#415

Post by nathan_long »

collyw wrote: post #332
varunom wrote: post #300

Earlier quoted context omitted.

I second this. My first task in any project is to design the whole data model based on current requirements and while designing it I think of the interfaces and how would they read and write data (to refine requirements). Writing views and actions/APIs on top of well-formed data model then becomes a breeze.

Its so un-agile (but it works).

Agile doesn't mean "don't gather requirements or plan anything." It just means that you evaluate your results frequently and maybe change course, instead of waiting until the end when you're "done".

Re: MongoDB queries don’t always return all matching documents

#416

Post by nathan_long »

rgo wrote: post #126

Everytime I hear arguments for going back to relational databases, I remember all the scalability problems I lived through for 15 years in relational hell before switching to Mongo. The thing about relational databases is that they do everything for you. You just lay the schema out (with ancient E-R tools maybe) load your relational data, write the queries, indexes, that's it. The problem was scalability, or any toug…

Suppose you're writing a system for vacation rentals. Here are two data consistency rules: 1) only one user may have a given email address 2) a given property can't have two overlapping rentals.

Postgres can enforce both, in a way that's not subject to race conditions. 1) Is a unique constraint and 2) is an exclusion constraint.

As far as I'm aware, the only way for application code to do this would be for it to 1) lock an entire table/collection of data 2) do a query to see what data is there 3) check the new data against the existing 4) write new data 5) unlock the database.

Postgres probably has to do the same thing conceptually, but by using indexes and running the checks in C on the same machine where the data lives, it can do it very quickly.

Are you saying you have a better solution?

Re: MongoDB queries don’t always return all matching documents

#417

Post by hk__2 »

matwood wrote: post #359
hk__2 wrote: post #95

Earlier quoted context omitted.

It’s useful for prototyping. When you don’t know which schema you’ll end up using having an *SQL database is tedious because you have to do migrations every time you change the schema. Once you’re done prototyping you can switch to a better alternative.

Use SQLite and let the tool drop and recreate all the tables during prototyping. No migrations required.

SQLite can’t handle a large number of documents; I have ~40M of them.

Re: MongoDB queries don’t always return all matching documents

#418

Post by hk__2 »

virmundi wrote: post #323
hk__2 wrote: post #95

Earlier quoted context omitted.

It’s useful for prototyping. When you don’t know which schema you’ll end up using having an *SQL database is tedious because you have to do migrations every time you change the schema. Once you’re done prototyping you can switch to a better alternative.

I guess this is a question of how usefuly, deployable the prototype shoukd be. Why not just have an in memory object cache, literally a hashmap, for your dal? If you're composing app level code, you don't need to know what the backend does to your data. You could even create a simple method to populate the data at app boot in your in the dev profile. When you figure out the storage requirements and finalized model, b…

> You could even create a simple method to populate the data at app boot in your in the dev profile

I have ~40M documents, and it’s still a subset of what I’ll manage in production. It takes a few hours to load them in the database.

Re: MongoDB queries don’t always return all matching documents

#419

Post by hk__2 »

collyw wrote: post #251
hk__2 wrote: post #95

Earlier quoted context omitted.

It’s useful for prototyping. When you don’t know which schema you’ll end up using having an *SQL database is tedious because you have to do migrations every time you change the schema. Once you’re done prototyping you can switch to a better alternative.

I have never managed to build a prototype that didn't end up being used in a production setting. I am trying to remember if I ever built a prototype that got rewritten. Probably not.

It probably won’t get rewritten but the DB-specific code is in one module so it’s easy(-ish) to swap the underlying DB.
Post reply on HN