MongoDB queries don’t always return all matching documents • Hacker News Board

Re: MongoDB queries don’t always return all matching documents

#401

Post by xapata » Wed, Jun 08, 2016, 11:12 PM UTC

erez wrote: post #270

xapata wrote: post #160

Earlier quoted context omitted.

> used mongo for years before we got our shit together. That's actually a legit use case. Use MongoDB while you get your shit together. I use global variables while I'm noodling around in code. Eventually I refactor.

I think this is a recipe for disaster. First, there are basic things that you should do from the get-go, e.g. not using globals. Second, the problem with "eventually I'll do it right" is that by that time, your stuff is out in the open, used by clients and heavily depended upon, and you have no way of refactoring. A company that uses a bad piece of technology will suffer many years before they could replace it.

> basic things ... not using globals

Depends on the language, I suppose. I'm more productive with Python when I write everything procedurally and refactor into functions, classes, etc. every dozen lines or so. It's more fun than writing UML diagrams (and seems to produce better code, too!).

Or do you think so long that your head aches and your colleague Hephaestus splits you open to find a fully-formed cooperative multiple inheritance hierarchy?

Re: MongoDB queries don’t always return all matching documents

#402

Post by xapata » Wed, Jun 08, 2016, 11:13 PM UTC

arctor wrote: post #233

xapata wrote: post #160

Earlier quoted context omitted.

> used mongo for years before we got our shit together. That's actually a legit use case. Use MongoDB while you get your shit together. I use global variables while I'm noodling around in code. Eventually I refactor.

The problem is, in a great portion of real world projects "eventually" never comes and there's just no time for any major refactoring or replacing technologies since you are too busy implementing the feature that was needed two weeks ago.

I'm not talking about major refactoring, but the kind of refactoring that I do every few minutes.

Re: MongoDB queries don’t always return all matching documents

#403

Post by xapata » Wed, Jun 08, 2016, 11:21 PM UTC

ckastner wrote: post #265

xapata wrote: post #160

Earlier quoted context omitted.

> used mongo for years before we got our shit together. That's actually a legit use case. Use MongoDB while you get your shit together. I use global variables while I'm noodling around in code. Eventually I refactor.

I don't think that is an apt comparison. Replacing your database backend, at the minimum, usually requires a massive migration of data, and possibly even changes to your entire architecture. A refactoring does not change behavior, and can be perfomed in minor -- and in your example of a global variable, perhaps even trivial -- increments. edit: typo

Sure, there's a continuum of refactoring, from trivial to complete re-write.

Any time the data schema(s) change, you need to migrate. I'll bet that even when sticking with the same database flavor you'll need to migrate a handful of times over the first few months. Requirements change, blah, blah. After the first couple migrations, you refactor to make that less painful. Eventually it might get to the point that your persistence layer is fairly abstracted and you can change databases without ripping apart everything else. Doesn't happen with every project, but sometimes.

Re: MongoDB queries don’t always return all matching documents

#404

Post by Idontreddit » Thu, Jun 09, 2016, 12:47 AM UTC

Lazare wrote: post #124

spriggan3 wrote: post #76

Earlier quoted context omitted.

> Why on earth would you use a database that makes so little in the way of guarantees about what results you get from it? Because some people can't stand having to work with SQL,migrations,schema and constraints, it's as simple as that ( That's not my opinion,that's just the rational behind MongoDB). Even if you use Postgres with the Json column type, you still need to write SQL queries and schemas. In the context of…

> Because some people can't stand having to work with SQL,migrations,schema and constraints The thing is, if you actually try and write an app using MongoDB, you will rapidly find that you: 1) Have migrations (except they're going to be some scary ad hoc nodejs script that loop through your document store and modify fields on the fly). 2) Have schemas (except they'll be implicit and undocumented) 3) Constraints (exce…

> If you're EVER going to read the data back and do anything with it, it has a schema.

You are giving the NoSQL crowd too much credit. Some abominations have no recognizable schema at all. The data store will just contain arbitrary dump of data which different developers decided their "schema" should be. The number of "columns" will vary, the "columns" will have arbitrary formats, so on and so forth.

If one developer decided to separate name into "first: John", "last: Doe", you will have that. If another decided to have "name: John Doe". That's what will be there. If one developer decided social security should be "SSN: 123-45-6789" and another decided it should "SSN: 123456780", well you are going to have fun cleaning up the data at the business or even application layer.

But that's not even the big issue with MongoDB. It's their lack of ACID compliance!

Re: MongoDB queries don’t always return all matching documents

#405

Post by Idontreddit » Thu, Jun 09, 2016, 1:07 AM UTC

im_down_w_otp wrote: post #16

Said it before, will say it again... "MongoDB is the core piece of architectural rot in every single teetering and broken data platform I've worked with." The fundamental problem is that MongoDB provides almost no stable semantics to build something deterministic and reliable on top of it. That said. It is really, really easy to use.

Yes it is easy to use. But too bad it also fails "transactions" silently so that you don't even know if your changes were "committed" or not. Don't worry, it only happens every once it a while so it's not a big deal...

Unless you are coinbase or an organization that deals with money/bitcoins/etc and you need ACID compliant transactions so that "debits/credits" don't just magically disappear.

When the bitcoin craze was going crazy, coinbase had all kinds of problems due to their mongodb backend.

Re: MongoDB queries don’t always return all matching documents

#406

Post by ahachete » Thu, Jun 09, 2016, 9:37 AM UTC

nimrody wrote: post #248

ahachete wrote: post #51

Strongly biased comment here, but hope its useful. Have you tried ToroDB ( https://github.com/torodb/torodb )? It still has a lot of room for improvement, but it basically gives you what MongoDB does (even the same API at the wire level) while transforming data into a relational form. Completely automatically, no need to design the schema. It uses Postgres, but it is far better than JSONB alone, as it maps data to re…

How does ToroDB handles sharding across multiple instances?

Right now ToroDB handles sharding at the backend (RDBMS) level, with those dbs that support that. There's currently a Greenplum-based backend on the works, that obviously handles sharding by itself. Also CitusDB is on the roadmap.

At a later release, we also plan to natively support MongoDB's sharding protocol.

Re: MongoDB queries don’t always return all matching documents

#407

Post by chris_wot » Thu, Jun 09, 2016, 10:48 AM UTC

hitr wrote: post #200

eloff wrote: post #41

Earlier quoted context omitted.

As a guy who works on ACID database internals, I'm appalled that people use MongoDB. You want a document store? Use Postgres. Why on earth would you use a database that makes so little in the way of guarantees about what results you get from it? I think most people have really low load and concurrency, so things seem to work. When things get busier you're in for a world of pain. Look I get that's it's easy to use and…

A startup which I have helped architect back end uses mongodb for everything. before starting the project I have requested the CTO not to use mongo as it was not a right fit.Basically they needed more of a relational stuff.The CTO chose mongo because he was thinking every startup uses it and why not us. Now they are suffering as they need ACID and relational features. They want to rewrite to postgres but they are hea…

Sounds like a startup with a very short timeframe.

Re: MongoDB queries don’t always return all matching documents

#408

Post by dkersten » Thu, Jun 09, 2016, 11:36 AM UTC

SmirkingRevenge wrote: post #219

m_mueller wrote: post #118

Earlier quoted context omitted.

So my question is then: Why not use CouchDB instead? I don't see what Mongo gives you over that and CouchDB is at least dependable and predictable in its operation.

I really like Couch, I wish it had more adoption than it seems to have and that its ecosystem was more mature than it seems to be... and that javascript wasn't its first class citizen. But its a really cool database (though I'm partial to rethinkdb now)

though I'm partial to rethinkdb now

Me too. RethinkDB is my document database of choice these days. In my experience, its proven to be reliable and fast and the development team very responsive and helpful. They also seem quite mindful when it comes to new features and will delay things for years (eg auto-failover, which they now support but it took a while) if rushing it would impact quality.

That's what I want from a database: first and foremost it must be solid and not lose my data. Everything else (including high availability) can come after.

Re: MongoDB queries don’t always return all matching documents

#409

Post by mikekchar » Thu, Jun 09, 2016, 12:11 PM UTC

fiatjaf wrote: post #372

mikekchar wrote: post #172

Earlier quoted context omitted.

I'm not the original poster, but I can give you some of my limited experience with CouchDB from an application I inherited. The original idea for the project still seems like a good idea to me. Basically they wanted to record events that came into the system and store them in a write only ledger. Then they wanted to version every change so that you have an audit trail. Finally they wanted to be able to create views o…

one of the things I want to try (but have only done some preliminary trials with) is to keep the concept of the write only ledger, but to replicate the db into several views of the data (some with severely restricted content). How is that different from the current concept of CouchDB views? You meant to replicate the DB to various different places and use the same CouchDB views from there? Or you meant something like…

Yes, the latter. One of the main problems I've seen is that indexing things that you will never query is both expensive in time and space. Also it's amazing how many views tend to have exactly the same data, only sorted differently. And the reason to sort it differently is because you only want to work on a subset of the data, but you can only restrict the query in a contiguous section of keys.

An example of this might be that you have a large number of daily reports. They all need different aspects of the data, but you end up writing views that sort by date and then collate the result in the server. So you end up maintaining an index for data that you will never query again and you are doing lots of extra processing merging the data after the query. Much better to replicate one day's worth of data to a new db every evening (possibly setting up a continuous replication to keep it up to date) and then add views on that db to do what you want. Like I said a full replication of a million documents takes about 10 minutes, so it's a reasonable thing to do.

Re: MongoDB queries don’t always return all matching documents

#410

Post by ahachete » Thu, Jun 09, 2016, 2:52 PM UTC

dkersten wrote: post #304

chris_wot wrote: post #282

Earlier quoted context omitted.

So get them to use ToroDB.

Never heard of ToroDB. Just checked out the website and it looks interesting, however the tagline "The first NoSQL and SQL database" is untrue. At least OrientDB has had both schema+schema-free and SQL + NoSQL querying interfaces. That is, you can optionally supply a schema for your documents. IIRC you could choose either schemaless, schema or mixed (where mixed allows fields not in the schema to exist as schemaless…

We have rectified the web site. No longer says "first", just states now the fact that its open source.

Thanks!