Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Migrated Away From MongoDB

svs.io

Post reply on HN
Print view View original

181–190 of 213 posts

Re: Why I Migrated Away From MongoDB

#181

Post by [deleted] »

gaius wrote: post #175
jeremyjh wrote: post #125

Earlier quoted context omitted.

Just guys like you or me except they build systems with billions of page views a month.

At my last job, we averaged 60k pages/sec, on what the NoSQL guys would call a "legacy" database, and there was plenty of headroom. It's not rocket science, just engineering. Companies like Google love to weave a mystique around what they do, it's in their interests after all, to convince their investors that what they do is magic. But I'm from back in the day when we were the same way about millions of pages per mon…

[deleted]

Re: Why I Migrated Away From MongoDB

#182

Post by [deleted] »

gaius wrote: post #175
jeremyjh wrote: post #125

Earlier quoted context omitted.

Just guys like you or me except they build systems with billions of page views a month.

At my last job, we averaged 60k pages/sec, on what the NoSQL guys would call a "legacy" database, and there was plenty of headroom. It's not rocket science, just engineering. Companies like Google love to weave a mystique around what they do, it's in their interests after all, to convince their investors that what they do is magic. But I'm from back in the day when we were the same way about millions of pages per mon…

[deleted]

Re: Why I Migrated Away From MongoDB

#183

Post by einhverfr »

gaius wrote: post #175
jeremyjh wrote: post #125

Earlier quoted context omitted.

Just guys like you or me except they build systems with billions of page views a month.

At my last job, we averaged 60k pages/sec, on what the NoSQL guys would call a "legacy" database, and there was plenty of headroom. It's not rocket science, just engineering. Companies like Google love to weave a mystique around what they do, it's in their interests after all, to convince their investors that what they do is magic. But I'm from back in the day when we were the same way about millions of pages per mon…

On good hardware there were PostgreSQL instances running a billion queries a day back in the 9.0 days. Often these are actual accounting apps where reporting matters and so NoSQL would be a very poor fit. For example the French government uses it to distribute welfare program dispersements. The Wisconsin Courts also uses PostgreSQL at loads in the billion per day range. I know there are larger instances out there.

Now a days we are talking about thousands of concurrent users and up to 350k reads/sec on high end machines.

Re: Why I Migrated Away From MongoDB

#184

Post by einhverfr »

einhverfr wrote: post #170
stickfigure wrote: post #169

Earlier quoted context omitted.

I'm guessing you're a big fan of stored procedures too. Good luck.

I'm guessing you're a big fan of stored procedures too. In certain times and places. However, I am not a fan of the idea that all db access goes through stored procedures. LedgerSMB will eventually do everything through stored procs but the reason is to make sure that all the logic is available to other apps, not to require that everyone else do all access through the stored procs. Also our stored procs are designed…

Too late to edit but it is worth noting that stored procedures are also very much all or nothing in terms of quality. Well-writen SP's are a delight to work with. Horribly written ones are infuriating. It's not like other environments where the quality of the code can degrade gracefully.

Re: Why I Migrated Away From MongoDB

#185

Post by einhverfr »

rbranson wrote: post #46
stickfigure wrote: post #43

Earlier quoted context omitted.

it's best just to start with a SQL database and go from there. This is bad advice. It's best to understand your problem domain and use the tools that are most appropriate. You see a lot of two types of posts on HN: * "I picked a NoSQL database for a problem domain with a better relational fit." Those posts look like this one. * "I picked an RDBMS for a problem domain with a better NoSQL fit." Those posts are usually…

Given the context of the OP, a brand new project or startup, my advice still stands true.

Part of the reason though is it is easier to get an idea of what you need from a NoSQL database and what you'd be giving up if you are already on an RDBMS, right?

I think the advice is right on and I see NoSQL databases as very much useful adjuncts to the relational systems, not as replacements.

Re: Why I Migrated Away From MongoDB

#186

Post by gaius »

einhverfr wrote: post #183
gaius wrote: post #175

Earlier quoted context omitted.

At my last job, we averaged 60k pages/sec, on what the NoSQL guys would call a "legacy" database, and there was plenty of headroom. It's not rocket science, just engineering. Companies like Google love to weave a mystique around what they do, it's in their interests after all, to convince their investors that what they do is magic. But I'm from back in the day when we were the same way about millions of pages per mon…

On good hardware there were PostgreSQL instances running a billion queries a day back in the 9.0 days. Often these are actual accounting apps where reporting matters and so NoSQL would be a very poor fit. For example the French government uses it to distribute welfare program dispersements. The Wisconsin Courts also uses PostgreSQL at loads in the billion per day range. I know there are larger instances out there. No…

Yes there is a lurking iceberg of things that aren't in the public eye like Facebook or Google, and that's where the really intense and interesting stuff is happening. I wonder what the guys at Visa make of all the hype around these websites, when they were doing these volumes all along...

At said 60k pages/sec job, you know who we looked up to for databases? Starbucks. Walmart. McDonalds. 'Cos we'd seen what they do, and anyone who thinks Twitter is impressive, well it'd blow their minds.

Re: Why I Migrated Away From MongoDB

#187

Post by einhverfr »

gregjor wrote: post #69
bunderbunder wrote: post #55

Earlier quoted context omitted.

SQL databases (except for SQLite) are almost completely interchangeable because they are all based on the same relational model and they all implement the ANSI SQL standard with only minor deviations. They only all implement the ANSI SQL standard for certain values of "ANSI SQL standard." Those values being SQL-89 or perhaps SQL-99. Beyond that it's a mess. SQL Server didn't get around to doing a really good job with…

I agree that there are numerous inconsistencies. I just have a different definition of "trivial cases" than you do. The kinds of things you mention -- real issues that come up in real applications -- fall outside of what most databases are doing in the kinds of web apps that I work on. I haven't had to deal with Oracle vs. SQL Server bulk load problems or Oracle's unique implementation of NULLs in years. Not waving t…

>"Not waving them away, just saying that these kinds of issues are not all that common because most real-world databases are trivial cases."

If this is the case, then, it's only because database application developers go out of their way to write compatible code. Things like upserts are not trivially compatible between vendors, and so are avoided by people trying to write portable applications.

Similarly, let's look at SQL 2003 structure data type inheritance and how this is implemented in Informix, Oracle, DB2, and PostgreSQL. Ok, so it isn't in PostgreSQL, and Informix allows tables to be inherited as well as types, and will even happily return result sets with different numbers of columns per row (google "Informix jagged rows")... And Oracle and DB2 look pretty close in this way until you get under the hood.

Re: Why I Migrated Away From MongoDB

#188

Post by einhverfr »

jamesli wrote: post #60
rbranson wrote: post #14

I'm no fan of MongoDB, but this same advice goes for any NoSQL data store. I am an Apache Cassandra contributor and community MVP, but my advice stays the same: it's best just to start with a SQL database and go from there. Read some books and learn it well: the "SQL Cookbook" from O'Reilly is great, and so is "The Art of SQL." Premature optimization continues to be the root of all evil.

I agree that relational databases are a safer bet after you study the business domains, consider the pros and cons of relational databases vs NoSQL, and there are no clear winners. Disagree to the books recommended. SQL is only a query language, not the database itself. It definitely should be part of the consideration. Understanding how database engines works under the hood is more important in terms of performance…

This is true. Robert Haas ran a couple of posts a while ago about differences between InnoDB and PostgreSQL. He basically pointed out that in InnoDB, the table is contained in the primary key index, while in PostgreSQL, it is an indexed heap of pages. Moreover InnoDB does not support traversing the table in physical order, so sequential scans of a large table are ghastly performance-wise.

What this means is that optimizations appropriate for InnoDB (index everything you search on) are very poor for PostgreSQL and vice versa. In PostgreSQL you index only a few columns, typically join conditions and a few select search columns, and the combination of the operating system and database will do the rest. In short, InnoDB only does index scans (and the table is a special index). This makes some things perform better (primary key lookups), but many other things perform worse. Moreover optimizing for one does not mean optimizing for the other....

Re: Why I Migrated Away From MongoDB

#189

Post by einhverfr »

gaius wrote: post #186
einhverfr wrote: post #183

Earlier quoted context omitted.

On good hardware there were PostgreSQL instances running a billion queries a day back in the 9.0 days. Often these are actual accounting apps where reporting matters and so NoSQL would be a very poor fit. For example the French government uses it to distribute welfare program dispersements. The Wisconsin Courts also uses PostgreSQL at loads in the billion per day range. I know there are larger instances out there. No…

Yes there is a lurking iceberg of things that aren't in the public eye like Facebook or Google, and that's where the really intense and interesting stuff is happening. I wonder what the guys at Visa make of all the hype around these websites, when they were doing these volumes all along... At said 60k pages/sec job, you know who we looked up to for databases? Starbucks. Walmart. McDonalds. 'Cos we'd seen what they do…

Have you looked at Postgres-XC yet? It's Teradata-style clustering for PostgreSQL.....

Re: Why I Migrated Away From MongoDB

#190

Post by einhverfr »

gregjor wrote: post #75
blaines wrote: post #70

Earlier quoted context omitted.

So do you believe there is a use case for a document-oriented database? I feel like your comment writes off a huge swath of useful technology. I said this below, but I'll say it again. Data is malleable, and writing apps to fit around any datastore seems wrong. I write applications to fulfill their use case. When the needs of the application change, so could it's database (or other dependencies). http://gigaom.com/cl…

I don't know if there's a use case. I have never had one come up in my own projects. I do sometimes need to manage a large number of documents (unstructured data) but I have found the Unix file system sufficient for those, maybe augmented by metadata in the database. I've worked on big applications with big databases where DB operations and aggregations started to get too slow but so far I or someone smarter on the t…

I suspect that a lot of NoSQL folks come to NoSQL from MySQL.

As for this part:

>"When you have multiple applications accessing the same database, each with their own implementation of consistency rules and their own internal idea of the data schema, come back and explain how that is more fun than using an RDBMS and being able to sleep nights."

The obvious answer from the MySQL/NoSQL folks is "What? That's crazy! Multiple apps hitting the same db? Are you nuts???"

See my blog post at http://ledgersmbdev.blogspot.com/2012/09/or-modelling-interl... for more on this mentality.

Post reply on HN