Why I Migrated Away From MongoDB • Hacker News Board

Re: Why I Migrated Away From MongoDB

#161

Post by einhverfr » Tue, Sep 18, 2012, 3:20 AM UTC

lmm wrote: post #19

jaequery wrote: post #9

I've ran into similar issue as you described. Something that can be done so simple and quickly in SQL, was bewilderingly difficult to do in mongo. The schema-less database approach also seems attractive at first but updating your data whenever your "app schema" changes starts to become a pain real quick. Now I can't really live w/o having a schema first, it actually saves you a lot more time in the long run (even sho…

You still need to update your data when doing schema changes under SQL, and you have a lot less control over the process. And you can do anything to your data without a schema, you just need to build your app as a service that provides access to it. IME SQL schemas do more harm than good; usually you end up with a schema that's subtly weaker than what's actually valid for your application, and the difference between…

IME SQL schemas do more harm than good

Completely disagree here. The basic tradeoff is between flexible input and flexible output. Without a rigid schema, ad hoc reporting is impossible because you don't have an ability to articulate reporting criteria. I.e. no declarative schema means no declarative reporting query.

I suppose that's ok as long as you never need to report on anything..... Might work....

Re: Why I Migrated Away From MongoDB

#162

Post by nerd_in_rage » Tue, Sep 18, 2012, 3:27 AM UTC

lttlrck wrote: post #155

nerd_in_rage wrote: post #131

Earlier quoted context omitted.

What? It's the absolute truth. If someone's wall post or tweet goes missing, it's totally inconsequential. "Social graph" - LOL.

You think FB doesn't consider durability and consistency essential? With almost one billion users?! Wow. Whether it is consequential or not is completely beside the point. As is whether you understand the "social graph".

Eventual consistency, yes. What I'm saying is if a few pics disappear for a half hour while things resolve themselves, nobody is going to care... or likely notice.

It's not a bank.

By the way, I believe social networking is mostly bullshit, which is my "LOL" at the "social graph".

Re: Why I Migrated Away From MongoDB

#163

Post by einhverfr » Tue, Sep 18, 2012, 3:34 AM UTC

stickfigure wrote: post #150

einhverfr wrote: post #144

Earlier quoted context omitted.

It's best to understand your problem domain and use the tools that are most appropriate. Sure. The big problem though is that the problem domain of RDBMS's is ad hoc reporting which is pretty universal. rom my experience, you should be leaning towards NoSQL systems (of which there are many, each suited to different use cases) when you have very large scaling needs (in terms of dataset and qps), heavily polymorphic da…

Even for ad-hoc reporting, the relational model has limits. For some requirements specialized structures are a better choice; thus we have mdolap. RDBMSes were designed for an era of spinning platters, data volumes in the megabytes-to-gigabytes, and small volumes of long-running ad-hoc queries. It's remarkable how far we've been able to push a general-purpose design but it's starting to burst the seams. Do you want t…

Even for ad-hoc reporting, the relational model has limits. For some requirements specialized structures are a better choice; thus we have mdolap.

But mdolap is far easier to work with if your feed is from a relational model. Otherwise you are left with a lot of effort getting the data transformed into an appropriate model.

Relational math + fixed schemas means flexible output.

I don't think that overlap is as big as you do.

Any line of business app had better start with a relational model then, as would any case where you are selling business intelligence. The other data stores work best as adjuncts to, rather than replacements for, a relational store.

Edit: Also it occurs to me that MDOLAP is only partly ad hoc. You have to set up your cubes ahead of time and that means deciding on what you are going to report. I imagine that adding a new reporting dimension to a large data set would be painful.

Re: Why I Migrated Away From MongoDB

#164

Post by josephcooney » Tue, Sep 18, 2012, 3:38 AM UTC

redler wrote: post #35

digiDoc is all about converting paper documents like receipts and business cards into searchable database, and so a document database seemed like a logical fit(!). It looks like this single initial assumption is where things started going wrong: conflating the pieces of paper that happen to be called "documents" in the real world with the concept of a "document" in the context of a system like MongoDB.

It's an easy mistake to make - to assume two things with the same name might be similar. Especially when paper documents have been called thus for a very long time.

It seems like another instance where all the good names were taken. http://jcooney.net/post/2012/04/03/Transaction-Argument-Clas...

Re: Why I Migrated Away From MongoDB

#165

Post by einhverfr » Tue, Sep 18, 2012, 3:45 AM UTC

stickfigure wrote: post #160

einhverfr wrote: post #145

Earlier quoted context omitted.

The basic tradeoff is between flexible input and flexible output. I do think it is sound advice to push the relational database model (this overlaps but is not identical to the relational algebra model) to the breaking point before going elsewhere. If you don't do data validation you can't do ad hoc data transformation later reliably. Strict schemas are always an investment in the future for that reason. > Polymorphi…

JSON and XML types give you all the pain of a NoSQL system with none of the benefits of an RDBMS. No thanks. My tools elegantly map database fields to my objects; doing this with serialization formats is incredibly crude and brittle. I do it for some edge cases already; I am looking forward to removing that code. A schemaless datastore does not mean a schemaless app. Furthermore, the application layer is a far better…

Typically if you have some polymorphic data attached to some other data, a hybrid approach works. After all, you can't report on what you can't easily quantify.

My tools elegantly map database fields to my objects; doing this with serialization formats is incredibly crude and brittle.

Build tools to do it.

A schemaless datastore does not mean a schemaless app.

That's actually a problem in most cases.

But there are many applications that are better suited to alternatives, especially with large-scale consumer-facing web apps. These threads annoy me; we should all be looking at new tools saying "wow, I wonder what this is good for" rather than smugly burning all the heretics that turn away from to the One True Idol.

Certainly that last part is true, but the best uses I have seen, including uses put forth by NoSQL proponents, have been as adjuncts to a relational approach, for example handling data transformation on the way in or out. Otherwise you end up with two fundamental problems:

1) How many other apps can your application support? (see https://news.ycombinator.com/item?id=4495377) Where is your public API? Basically you can do this in your db layer and/or your app layer, but you get a lot more flexibility by doing it in the db. With a schemaless app, you can't do it there without intimate knowledge of innards...

2) Strict schemas with validated input and a declarative query language buys you flexible output. Rigid schemas are thus an investment in the future. If your data is sufficiently polymorphic that output can't be flexible no matter how you do it, then I suppose that's not a huge issue. However for many cases where NoSQL is chosen wrongly it is a huge issue.

Re: Why I Migrated Away From MongoDB

#166

Post by einhverfr » Tue, Sep 18, 2012, 3:47 AM UTC

nerd_in_rage wrote: post #162

lttlrck wrote: post #155

Earlier quoted context omitted.

You think FB doesn't consider durability and consistency essential? With almost one billion users?! Wow. Whether it is consequential or not is completely beside the point. As is whether you understand the "social graph".

Eventual consistency, yes. What I'm saying is if a few pics disappear for a half hour while things resolve themselves, nobody is going to care... or likely notice. It's not a bank. By the way, I believe social networking is mostly bullshit, which is my "LOL" at the "social graph".

Eventual consistency however provides no guarantee of consistency unless it is backed by absolute local consistency on every node. Don't you run into CAP theorem problems otherwise?

Re: Why I Migrated Away From MongoDB

#167

Post by lmm » Tue, Sep 18, 2012, 3:57 AM UTC

einhverfr wrote: post #161

lmm wrote: post #19

Earlier quoted context omitted.

You still need to update your data when doing schema changes under SQL, and you have a lot less control over the process. And you can do anything to your data without a schema, you just need to build your app as a service that provides access to it. IME SQL schemas do more harm than good; usually you end up with a schema that's subtly weaker than what's actually valid for your application, and the difference between…

IME SQL schemas do more harm than good Completely disagree here. The basic tradeoff is between flexible input and flexible output. Without a rigid schema, ad hoc reporting is impossible because you don't have an ability to articulate reporting criteria. I.e. no declarative schema means no declarative reporting query. I suppose that's ok as long as you never need to report on anything..... Might work....

I'm not saying don't have a rigid schema, just don't enforce it at the storage layer. If you're doing an ad-hoc report then you wouldn't have indexes in place for it in the SQL case, so I don't see how it's any worse or harder in mongodb.

Re: Why I Migrated Away From MongoDB

#168

Post by einhverfr » Tue, Sep 18, 2012, 4:00 AM UTC

lmm wrote: post #167

einhverfr wrote: post #161

Earlier quoted context omitted.

IME SQL schemas do more harm than good Completely disagree here. The basic tradeoff is between flexible input and flexible output. Without a rigid schema, ad hoc reporting is impossible because you don't have an ability to articulate reporting criteria. I.e. no declarative schema means no declarative reporting query. I suppose that's ok as long as you never need to report on anything..... Might work....

I'm not saying don't have a rigid schema, just don't enforce it at the storage layer. If you're doing an ad-hoc report then you wouldn't have indexes in place for it in the SQL case, so I don't see how it's any worse or harder in mongodb.

>If you're doing an ad-hoc report then you wouldn't have indexes in place for it in the SQL case, so I don't see how it's any worse or harder in mongodb.

Only true in a case where you ahve to index everything you might want to search on, like with InnoDB. In PostgreSQL all you really need are your foreign key indexes and a couple (if that) of criteria indexes and you are good. That's more of an InnoDB limitation than a relational limitation. Basically InnoDB tables are primary key indexes and they can only be traversed in key order, not physical order, so sequential scans are painful....

Re: Why I Migrated Away From MongoDB

#169

Post by stickfigure » Tue, Sep 18, 2012, 4:04 AM UTC

einhverfr wrote: post #165

stickfigure wrote: post #160

Earlier quoted context omitted.

JSON and XML types give you all the pain of a NoSQL system with none of the benefits of an RDBMS. No thanks. My tools elegantly map database fields to my objects; doing this with serialization formats is incredibly crude and brittle. I do it for some edge cases already; I am looking forward to removing that code. A schemaless datastore does not mean a schemaless app. Furthermore, the application layer is a far better…

Typically if you have some polymorphic data attached to some other data, a hybrid approach works. After all, you can't report on what you can't easily quantify. My tools elegantly map database fields to my objects; doing this with serialization formats is incredibly crude and brittle. Build tools to do it. A schemaless datastore does not mean a schemaless app. That's actually a problem in most cases. But there are ma…

I'm guessing you're a big fan of stored procedures too.

Good luck.

Re: Why I Migrated Away From MongoDB

#170

Post by einhverfr » Tue, Sep 18, 2012, 4:07 AM UTC

stickfigure wrote: post #169

einhverfr wrote: post #165

Earlier quoted context omitted.

Typically if you have some polymorphic data attached to some other data, a hybrid approach works. After all, you can't report on what you can't easily quantify. My tools elegantly map database fields to my objects; doing this with serialization formats is incredibly crude and brittle. Build tools to do it. A schemaless datastore does not mean a schemaless app. That's actually a problem in most cases. But there are ma…

I'm guessing you're a big fan of stored procedures too. Good luck.

I'm guessing you're a big fan of stored procedures too.

In certain times and places. However, I am not a fan of the idea that all db access goes through stored procedures. LedgerSMB will eventually do everything through stored procs but the reason is to make sure that all the logic is available to other apps, not to require that everyone else do all access through the stored procs.

Also our stored procs are designed to be discoverable by the application and therefore allow for looser coupling between the app and sproc than is typical.