Live data from Hacker News

A Year with MongoDB

blog.engineering.kiip.me

121–130 of 153 posts

Re: A Year with MongoDB

#121
post #94
post #78

Earlier quoted context omitted.

It's the same concept, but it's very different in practice. For example, adding a field with a default value in postgres will lock the table during the migration, which may be killer. If you use postgres for big data sets, what you'll end up implementing for your migrations looks a lot like what schemaless gives you for free.

If you add a default value it locks the table and re-writes it. However, if you don't add a default value postgres will perform the operation instantly. Old rows will be untouched, and new values will be added with the shoe_size of either NULL or whatever you set it to be... IE exactly the same outcome and performance as the MongoDB case mentioned above. Adding a field in postgres while setting a default value would…

Thank you very much for this quick tip. I've unfortunately only used MySQL at scale, and it most definitely grinds away for hours when ALTERing, default value or not.

It looks like MySQL might be giving everyone in the RDBS world a bad rap.

Re: A Year with MongoDB

#122

Earlier quoted context omitted.

MongoDB is successful because of more than just marketing. It has great tool support, decent documentation, books and is accessible. Plus the whole transition from MySQL concept makes it easy to grab onto.

That all supports the marketing effort. Mongo is optimized for a good out-of-the-box experience for developers. It is basically the MySQL model -- hook developers first, fix the fundamentals later. Caveat emptor.

Exactly, hook them in, so they question whether or not to deal with the problems when it falls on its face.

Re: A Year with MongoDB

#123
post #94

Earlier quoted context omitted.

If you add a default value it locks the table and re-writes it. However, if you don't add a default value postgres will perform the operation instantly. Old rows will be untouched, and new values will be added with the shoe_size of either NULL or whatever you set it to be... IE exactly the same outcome and performance as the MongoDB case mentioned above. Adding a field in postgres while setting a default value would…

> Now I'm not anti-MongoDB, I'm just saying you shouldn't give it credit for something that relational database do just fine. No, it can't. Sorry, it absolutely can't. > If you add a default value it locks the table and re-writes it. > However, if you don't add a default value postgres will perform the operation instantly. And in MongoDB you don't have to change anything in the DB (hence, no downtime). It's all in yo…

> I also can't use the old version of the DB, so my old backups just became more troublesome to recover.

Tangential to the topic, but this is important: every schema update MUST be considered part of the app / database / backend / system source code, and as such it should be properly and formally applied, versioned and backed up. In the simplest case, you can store a version number in the database itself, every update applies to a certain version # X and advances to another version # Y, and every update is stored in its own source file (updateX.sql for example).

Would be nice if DBMSs offered primitives specifically for schema version management, but I guess there could be many different use cases, and it's very easy to roll your own.

Re: A Year with MongoDB

#124
This is a pretty epic troll on MongoDB, and some of their points are important -- particularly global write lock and uncompressed field names, both issues that needlessly afflict large MongoDB clusters and will likely be fixed eventually.

However, it's pretty clear from this post that they were not using MongoDB in the best way. For example, in a small part of their criticism of "safe off by default", they write:

"We lost a sizable amount of data at Kiip for some time before realizing what was happening and using safe saves where they made sense (user accounts, billing, etc.)."

You shouldn't be storing user accounts and billing information in MongoDB. Perhaps MongoDB's marketing made you believe you should store everything in MongoDB, but you should know better.

In addition to that data being highly relational, it also requires the transactional semantics present in mature relational databases. When I read "user accounts, billing" here, I cringed.

Things that it makes total sense to use MongoDB for:

- analytics systems: where server write thorughput, client-side async (unsafe) upserts/inserts, and the atomic $inc operator become very valuable tools.

http://blog.mongodb.org/post/171353301/using-mongodb-for-rea...

- content management systems: where schema-free design, avoidance of joins, its query language, and support for arbitrary metadata become an excellent set of tradeoffs vs. tabular storage in an RDBMS.

http://www.mongodb.org/display/DOCS/How+MongoDB+is+Used+in+M...

- document management systems: I have used MongoDB with great sucess as the canonical store of documents which are then indexed in a full-text search engine like Solr. You can do this kind of storage in an RDBMS, but MongoDB has less administrative overhead, a simpler development workflow, and less impedance mismatch with document-based stores like Solr. Further, with GridFS, you can even use MongoDB as a store for actual files, and leverage MongoDB's replica sets for spreading those files across machines.

Is your data relational? Can you benefit from transactional semantics? Can you benefit from on-the-fly data aggregation (SQL aggregates)? Then use a relational database!

Using multiple data stores is a reality of all large-scale technology companies. Pick the right tool for the right job. At my company, we use MongoDB, Postgres, Redis, and Solr -- and we use them each on the part of our stack where we leverage their strengths and avoid their weaknesses.

This article reads to me like someone who decided to store all of their canonical data for an e-commerce site in Solr, and then complains when they realized that re-indexing their documents takes a long time, index corruption occurs upon Solr/Lucene upgrades, or that referential integrity is not supported. Solr gives you excellent full-text search, and makes a lot of architectural trade-offs to achieve this. Such is the reality of technology tools. What, were you expecting Solr to make your coffee, too?

Likewise, MongoDB made a lot of architectural tradeoffs to achieve the goals it set out in its vision, as described here:

http://www.mongodb.org/display/DOCS/Philosophy

It may be a cool technology, but no, it won't make your coffee, too.

In the end, the author writes, "Over the past 6 months, we've scaled MongoDB by moving data off of it. [...] we looked at our data access patterns and chose the right tool for the job. For key-value data, we switched to Riak, which provides predictable read/write latencies and is completely horizontally scalable. For smaller sets of relational data where we wanted a rich query layer, we moved to PostgreSQL."

Excellent! They ended up in the right place.

Re: A Year with MongoDB

#125

This is a pretty epic troll on MongoDB, and some of their points are important -- particularly global write lock and uncompressed field names, both issues that needlessly afflict large MongoDB clusters and will likely be fixed eventually. However, it's pretty clear from this post that they were not using MongoDB in the best way. For example, in a small part of their criticism of "safe off by default", they write: "We…

So, they made some mistakes, learned from them, and ended up in the right place.

Sounds like a great story for a blog post, that others might learn from as well.

Calling it a troll -- just because their mistakes involved mongo and their solution did not -- seems harsh.

Re: A Year with MongoDB

#126

This is a pretty epic troll on MongoDB, and some of their points are important -- particularly global write lock and uncompressed field names, both issues that needlessly afflict large MongoDB clusters and will likely be fixed eventually. However, it's pretty clear from this post that they were not using MongoDB in the best way. For example, in a small part of their criticism of "safe off by default", they write: "We…

So, they made some mistakes, learned from them, and ended up in the right place. Sounds like a great story for a blog post, that others might learn from as well. Calling it a troll -- just because their mistakes involved mongo and their solution did not -- seems harsh.

Fair enough, perhaps it's not quite a troll. And I am not trying to devalue the post or discussion.

But I'm finding that in this whole SQL vs. NoSQL debate, everyone is desperately seeking the "one database to store everything" -- rather than carefully evaluating trade-offs of systems before putting them into production.

The conclusion of the article suggests that new projects start with "PostgreSQL (or some traditional RDBMS) first", and then only switch to other systems "when you find them necessary". Wrong conclusion. Think about what you're building, and pick the right data store for your data.

Re: A Year with MongoDB

#127
post #86

Earlier quoted context omitted.

> isn't that where the schema belongs? [In the application] Well, unless you have, you know, multiple applications accessing said data. Then it's kind of important to keep it in sync, which is why RDBMS exist and operate the way they do. In my experience, on a long enough timeline, the probability of needing multi-application access for your data goes to 1.

build an api

A schema is essentially an API over the data.

For instance, if you have an API with the functions:

  item# = add_item(description, price, color)
  record_sale(item#, quantity, customer#)
So let's see what happens when you need to make a change. For instance, you might decide that an item could be discounted (a new feature); so you need to record the price at the time of sale:

  record_sale(item#, quantity, customer#, price=NULL)
and if the price is NULL, you default to the current item price.

But you already have sales data without a price, so you need to migrate it by copying the current item prices over.

And if one of three applications doesn't support the new API, you need to offer the old API as well and it would just use the default all of the time.

That sounds pretty much like a schema to me. What's the difference?

Re: A Year with MongoDB

#128

Earlier quoted context omitted.

So, they made some mistakes, learned from them, and ended up in the right place. Sounds like a great story for a blog post, that others might learn from as well. Calling it a troll -- just because their mistakes involved mongo and their solution did not -- seems harsh.

Fair enough, perhaps it's not quite a troll. And I am not trying to devalue the post or discussion. But I'm finding that in this whole SQL vs. NoSQL debate, everyone is desperately seeking the "one database to store everything" -- rather than carefully evaluating trade-offs of systems before putting them into production. The conclusion of the article suggests that new projects start with "PostgreSQL (or some traditio…

> everyone is desperately seeking the "one database to store everything" -- rather than carefully evaluating trade-offs

Agreed.

> Think about what you're building, and pick the right data store for your data.

I partially disagree. Most businesses adapt considerably over time, and data projects almost always expand as far as the engineering team can take them. Even small businesses have a lot of different kinds of data, all held in different systems and spreadsheets, and there is a lot of value in bringing that data together (often driven by accounting).

So, at the beginning, you don't know what your data is, you just have a vague idea (unless you are an early-stage startup with very specific data management needs).

(Aside: your queries are at least as important when choosing a data management system as your data).

Traditional RDBMSs have been designed and have evolved over a long period of time to offer pretty clear answers for the business data needs of most "normal" businesses. It makes perfect sense to start with a general solution, and try to pick out the special cases (e.g. "I need faster response on these queries") as you go.

That doesn't mean that traditional RDBMSs are the only way to make a general-purpose data management system. Maybe another architecture or model will come along that will prove superior; or maybe it's already here and it's just not mature enough.

But I would give very similar advice in most situations: start with a traditional RDBMS, and then pick out the special cases as needed. Not all cases, of course; a key-value system might be great for caching, or you might need to do analysis that's just not working out with SQL.

Re: A Year with MongoDB

#129

Earlier quoted context omitted.

build an api

It's a shame this comment is pithy, because I think it's dead-on. There times to integrate at the database level. But, the default should be single-application databases. The rationale is the same as the grandparent's rationale FOR database integration. The odds of needing to share data over time are 1. Given that shared belief, the problem with database integration is that MANY applications need to share facets of t…

> Thus, "build an api" is the best solution.

I think there's an asymmetry in your argument.

You are describing all of the problems with data management as though they were specific to schemas in a traditional RDBMS; but glossing over how "building an API" solves those same problems, and whether that method is better or worse.

In other words, "build an API" is the problem at hand, not the solution. A traditional DBMS provides a data definition language (DDL) to help solve the problem (schema design), which isn't a solution either, but it does offer a lot of direction. Saying "build an API" really gives me no direction at all.

Re: A Year with MongoDB

#130

Earlier quoted context omitted.

build an api

Agree with this 100%. In the past the way different applications integrate with each other was through sharing the same database tables. Hence the need to keep the schema synchronized across multiple applications. Nowadays applications should integrate through well-defined APIs and service interfaces. See this Martin Fowler short essay on database styles: http://martinfowler.com/bliki/DatabaseStyles.html

> Hence the need to keep the schema synchronized across multiple applications

Views are designed to avoid that problem.

What is the specific way you handle versioning in an API that is superior to versioning in the schema?

Post reply on HN