Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Year with MongoDB

blog.engineering.kiip.me

Post reply on HN
Print view View original

101–110 of 153 posts

Re: A Year with MongoDB

#101

Post by nivertech »

zzzeek wrote: post #17
mitchellh wrote: post #7

Earlier quoted context omitted.

[Note: I wrote the blog post] I'm not at all against horizontally scaling. However, I don't believe that horizontally scaling should be necessary doing a mere 200 updates to per second to a data store that isn't even fsyncing writes to disk. Think of it in terms of cost per ops. Let's just say 200 update ops per second is the point at which you need to shard (not scientific, but let's just use that as a benchmark sin…

typo in the numbers ? I can get your mongoDB number but not pg: >>> seconds_per_month = 60 * 60 * 24 * 30 >>> ops_per_month_pg = 1000 * seconds_per_month >>> ops_per_month_mg = 200 * seconds_per_month >>> 330.0 / ops_per_month_pg * 100 1.2731481481481482e-05 >>> 330.0 / ops_per_month_mg * 100 6.36574074074074e-05

You could have skipped all the calculations and just say, that 1000/200=5, I.e PostgreSQL 5 times more cost effective than MongoDB.

Re: A Year with MongoDB

#102

Post by zeeg »

pbiggar wrote: post #78
bradgessler wrote: post #71

Earlier quoted context omitted.

That's still a migration, its just incremental.

It's the same concept, but it's very different in practice. For example, adding a field with a default value in postgres will lock the table during the migration, which may be killer. If you use postgres for big data sets, what you'll end up implementing for your migrations looks a lot like what schemaless gives you for free.

This is easily solved in Postgres. You dont do alters with default, or non null values. Alters become fast [enough] and bearable, even at scale.

http://justcramer.com/2011/11/10/scaling-schema-changes/

Re: A Year with MongoDB

#103

Post by vsl »

latch wrote: post #90
jbellis wrote: post #14

Earlier quoted context omitted.

According to http://www.mongodb.org/display/DOCS/getLastError+Command it is still unsafe by default.

No, the journal file is turned on server-side and you'll get a journal append every 100ms (by default).

That's not what is meant by "safe" here. Safe is when you can be sure that the data were written (into the journal at least) by the time the write call in your code returns. Doing it up to 100ms later leaves a wide window open when the application believes the data are safely stored, while they are in fact in RAM only.

Re: A Year with MongoDB

#104

Post by cheald »

ericflo wrote: post #9
cheald wrote: post #8

Earlier quoted context omitted.

I think the article is a fair criticism, and I think your response is likewise fair. Mongo was built with horizontal scaling in mind, and to that end, it tends to suffer noticeably when you overload a single node. Things like the global write lock and single-threaded map/reduce are definitely problems, and shouldn't be pooh-pooh'd away as "oh, just scale horizontally". Uncompacted key names are a real problem, and a…

Mongo was not designed with horizontal scaling in mind. Riak, Cassandra, HBase, Project Voldemort...these are the projects that were designed with horizontal scaling in mind (as evidenced by their architectures.) But not Mongo.

Mongo's sort of a weird halfway point between "totally horizontal" stores like Riak or Cassandra, and "Hope you have a ton of RAM" stores like your traditional RDBMS setup. I think it's hard to look at it and say that it wasn't designed with horizontal scaling in mind, but it's totally fair to say that it wasn't designed purely for horizontal scaling, either.

Re: A Year with MongoDB

#105

Post by [deleted] »

mrkurt wrote: post #43
rdtsc wrote: post #10

> Safe off by default I think that is fixed now. But this the single most appalling design decision they could have made while also claiming their product was a "database". (And this has been discussed here before so just do a search if you will). This wasn't a bug, it was a deliberate design decision. Ok, that that would have been alright if they put a bright red warning on their front page. "We disabled durability…

"Safe" is a driver implementation detail, not really a server default change. The driver basically has to give the DB a command, then ask what just happened for "safe" writes. If the driver doesn't bother listening for the result, the database just does whatever it's going to quietly. That said, I really wish all the drivers issued the getLastError command after writes by default. It's the first thing we tell custome…

[deleted]

Re: A Year with MongoDB

#106

Post by fusiongyro »

quadhome wrote: post #93
matthewcford wrote: post #91

Earlier quoted context omitted.

build an api

It's a shame this comment is pithy, because I think it's dead-on. There times to integrate at the database level. But, the default should be single-application databases. The rationale is the same as the grandparent's rationale FOR database integration. The odds of needing to share data over time are 1. Given that shared belief, the problem with database integration is that MANY applications need to share facets of t…

> The single database ends up having a huge surface area trying to satisfy every application's needs.

This is, more or less, exactly what views are for.

> Thus, "build an api" is the best solution.

And you can do that within the database with stored procedures, perhaps even with the same language you would use in the front-end (depending). And look at the advantages you have:

- No implied N+1 issues because your API is too granular

- No overfetching because your API is too coarse

- No additional service layers needed

- All the information is in the right place to ensure data validity and performance

Let me be clear: I see these as two viable alternatives and different situations are going to determine the appropriate tool. I bring this up because I do think the NoSQL crowd overall has a very distorted and limited picture of what exactly it is RDBMSes provide and why. If people look underneath their ORMs, they may find an extremely powerful, capable and mature system under there that can solve lots of problems well—possibly (but I admit, not necessarily) even _their own_ problems.

Re: A Year with MongoDB

#107

Post by mtrn »

AdrianRossouw wrote: post #84
mtrn wrote: post #35

Earlier quoted context omitted.

Thanks for writing OrientDB! - I tried it, but I was pressed for time, so I needed something that more or less worked instantly for my requirements - which in the end was elasticsearch. TL; I researched MongoDB and OrientDB for a side-project with a bit heavy data structure (10M+ docs, 800+ fields on two to three levels). MongoDB was blazingly fast, but it segfaulted somewhere in the process (also index creation need…

i love ES, but i don't really feel comfortable with it as a primary datastore. We tend to use couchdb to write to, and ES to query against. It all happens automagically with a single shell command. I won't use ES on it's own, because I have experienced situations in the past where the dynamic type mapping functionality gets confused, ie: the first time it sees a field, it indexes it as an integer, but then one of the…

Thank you for your input. Had minor issues with dynamic mapping, too - but since the data is more or less just strings, I could circumvent ES' mechanism to infer datatype from value by simple using an empty default-mapping.js. I'll definitely give your approach a try.

Re: A Year with MongoDB

#108

Post by tsaixingwei »

matthewcford wrote: post #91
trimbo wrote: post #86

Earlier quoted context omitted.

> isn't that where the schema belongs? [In the application] Well, unless you have, you know, multiple applications accessing said data. Then it's kind of important to keep it in sync, which is why RDBMS exist and operate the way they do. In my experience, on a long enough timeline, the probability of needing multi-application access for your data goes to 1.

build an api

Agree with this 100%. In the past the way different applications integrate with each other was through sharing the same database tables. Hence the need to keep the schema synchronized across multiple applications. Nowadays applications should integrate through well-defined APIs and service interfaces. See this Martin Fowler short essay on database styles: http://martinfowler.com/bliki/DatabaseStyles.html

Re: A Year with MongoDB

#109

Post by raverbashing »

birken wrote: post #94
pbiggar wrote: post #78

Earlier quoted context omitted.

It's the same concept, but it's very different in practice. For example, adding a field with a default value in postgres will lock the table during the migration, which may be killer. If you use postgres for big data sets, what you'll end up implementing for your migrations looks a lot like what schemaless gives you for free.

If you add a default value it locks the table and re-writes it. However, if you don't add a default value postgres will perform the operation instantly. Old rows will be untouched, and new values will be added with the shoe_size of either NULL or whatever you set it to be... IE exactly the same outcome and performance as the MongoDB case mentioned above. Adding a field in postgres while setting a default value would…

> Now I'm not anti-MongoDB, I'm just saying you shouldn't give it credit for something that relational database do just fine.

No, it can't. Sorry, it absolutely can't.

> If you add a default value it locks the table and re-writes it. > However, if you don't add a default value postgres will perform the operation instantly.

And in MongoDB you don't have to change anything in the DB (hence, no downtime). It's all in your code.

Yes, I can add a field in PGSQL, then my ORM quits on me (because I can't have a optional field, of course). Or I can use hand written SQL at which point it adds a couple of months to my schedule.

Or I can use migrations + ORM like Ruby migrations or South (Django). They will blow up for anything slightly more complex.

I also can't use the old version of the DB, so my old backups just became more troublesome to recover.

And that's why Facebook and others don't user relational anymore. Sure, you can use MySQL or PG, but the real data is stored in a text or binary blob.

Re: A Year with MongoDB

#110

Post by amalag »

luca_garulli wrote: post #27

Hey, by reading all the bad things seems that OrientDB would fit better than MongoDB for them: - Non-counting B-Trees: OrientDB uses MVRB-Tree that has the counter. size() requires 0ns - Poor Memory Management: OrientDB uses MMAP too but with many settings to optimize it usage - Uncompressed field names: the same as OrientDB - Global write lock: this kills your concurrency! OrientDB handles read/write locks at segmen…

I have always been curious about OrientDB, but from what I saw it was very small and not backed by any commercial entity and it's usage was not widespread. Also Luca, you should in fairness write that you are the maintainer.
Post reply on HN