The genius and folly of MongoDB • Hacker News Board

Re: The genius and folly of MongoDB

#271

Post by dodyg » Tue, Oct 22, 2013, 11:01 AM UTC

ahoge wrote: post #220

JulianMorrison wrote: post #118

Earlier quoted context omitted.

Have you considered RethinkDB? For most of the advantages of MongoDB that don't specifically come from mmap and overwrite-in-place, it's an equal or better.

RethinkDB still doesn't support Windows. [1] Most developers who use Windows will just go with something which doesn't require a virtual machine. For example, MongoDB, CouchDB, OrientDB, Cassandra, and ArrangoDB work fine everywhere. [1] https://github.com/rethinkdb/rethinkdb/issues/1100

Did you try RavenDB which works quite well in Windows?

Re: The genius and folly of MongoDB

#272

Post by dodyg » Tue, Oct 22, 2013, 11:08 AM UTC

dlau1 wrote: post #151

Can someone comment on how mature rethinkdb is at the moment? I'm considering moving away from MongoDB before I have to implement what seems to be an incredibly complicated architecture to get it to scale on the level tens/hundreds of millions of documents.

RethinkDB doesn't have Geospatial index yet (if it's relevant to your needs)

https://github.com/rethinkdb/rethinkdb/issues/1158

Re: The genius and folly of MongoDB

#273

Post by philwelch » Tue, Oct 22, 2013, 1:34 PM UTC

nostrademons wrote: post #268

philwelch wrote: post #267

Earlier quoted context omitted.

If it takes you a year from persisting serialized data on the hard drive of your one server to using a real data store, you're fucked either way. As low as that half-life might be, that's no reason to make deliberately short-sighted engineering decisions to make it even worse, especially when all the quick and easy ways of shipping an MVP effectively preclude that strategy. You're gonna go through all the effort of s…

You understand that Hacker News uses precisely this persistence strategy (in-memory data structures with persistent state written to the filesystem on the hard disk of the server), and has been going on 6 years now? You also understand that most of the advice easily accessible on the Internet comes from people trying to sell you something, and so they have a vested interest in you adding many layers into your softwar…

If anything it's this fantasy of vertical scaling that's perpetuated by "people trying to sell you something". If you're going to go with "Hacker News does it, therefore it's okay", I guess that means it's sensible for any web app to use a Lisp dialect of their own invention implemented on top of Scheme, for URL's to be generated pseudorandomly and time out, and so forth.

Re: The genius and folly of MongoDB

#274

Post by ak217 » Tue, Oct 22, 2013, 4:39 PM UTC

pkolaczk wrote: post #237

ak217 wrote: post #82

Earlier quoted context omitted.

I'm curious, how do other DBMSs handle a master switch/other cluster updates? I'm familiar enough with mongos to know how it works but not what e.g. redis or postgres or mysql does.

The best is not to ever need it by using an architecture with no SPOF (even temporary). Master switch is a huge pain - there are simply too many nasty ways it can fail miserably. I'd stay away from databases needing it, if high availability is the primary concern.

OK, but that's a serious performance and functionality tradeoff.

Most modern architectures make the choice between having nodes serve as master for a subset of the data, and the increased cross-link bandwidth needs and reduced flexibility of a master-less system.

Re: The genius and folly of MongoDB

#275

Post by twic » Tue, Oct 22, 2013, 5:43 PM UTC

AlisdairO wrote: post #227

mdellabitta wrote: post #133

Earlier quoted context omitted.

Just ran into this link, which seems to describe how MVCC can sometimes not be enough. http://ronaldbradford.com/blog/understanding-innodb-mvcc-200...

Having read through it, I rather suspect that that's not a matter of writers blocking readers or the other way round, but instead a case of writers blocking writers - he's writing a lot of data to the table, and it's highly likely that InnoDB has escalated the lock to a table lock - which effectively prevents concurrent writes.

I don't believe that InnoDB escalates locks. Quoting from the fine manual:

http://dev.mysql.com/doc/refman/5.7/en/innodb-transaction-mo...

> InnoDB does locking on the row level and runs queries as nonlocking consistent reads by default, in the style of Oracle. The lock information in InnoDB is stored so space-efficiently that lock escalation is not needed: Typically, several users are permitted to lock every row in InnoDB tables, or any random subset of the rows, without causing InnoDB memory exhaustion.

Re: The genius and folly of MongoDB

#276

Post by nostrademons » Tue, Oct 22, 2013, 5:57 PM UTC

twic wrote: post #270

nostrademons wrote: post #211

Earlier quoted context omitted.

That's why you use a language or technology that's politically unfeasible for your rapid prototypes, like Clojure or Haskell, or...for that matter...MongoDB. ;-)

Someone tried that at my company. We now have a Clojure app in production. Brilliant.

Hey, it's a great way to get lesser-known languages into production!

I wonder if I'm going to be the one who finally switches Google Search over to Go, by way of a quick throwaway prototype...

Re: The genius and folly of MongoDB

#277

Post by twic » Tue, Oct 22, 2013, 6:02 PM UTC

JulianMorrison wrote: post #116

twic wrote: post #48

Earlier quoted context omitted.

Is it better than a relational database for that?

Most RDBMSs will, if you rewrite a field, write a fresh row and tombstone the old one, and clear it down in the next compaction. This is what MVCC means in practise: that the old version doesn't disappear while the new is being written. MongoDB by contrast will simply mmap that block of file, overwrite the contents, and fsync. Yes, this has obvious downsides.

I won't dispute the empirical finding that MongoDB and friends are faster at this than RDBMSs.

However, i'm not sure why this should be the case. You mention the complexity of updating a row in MVCC; sure, but all the database has to do before reporting success to the user is to write its intent to make this change to the transaction log (WAL in PostgreSQL, redo log in Oracle). The actual changes to the data files can be written back later on. The transaction log is a single stream being continuously written to disk, so that should be very fast.

MongoDB, on the other hand, is making scattered writes across its mmapped data files, which should be much slower. Except that of course it's probably doing this on a journalled filesystem, which is using exactly the same mechanism as the RDBMSs to provide fast, safe updates.

I'd be really interested to see how a simple update to a single field translates into actual writes to disk for PostgreSQL and MongoDB. If only i knew how to use strace!

Re: The genius and folly of MongoDB

#278

Post by ahoge » Thu, Oct 24, 2013, 10:31 AM UTC

dodyg wrote: post #271

ahoge wrote: post #220

Earlier quoted context omitted.

RethinkDB still doesn't support Windows. [1] Most developers who use Windows will just go with something which doesn't require a virtual machine. For example, MongoDB, CouchDB, OrientDB, Cassandra, and ArrangoDB work fine everywhere. [1] https://github.com/rethinkdb/rethinkdb/issues/1100

Did you try RavenDB which works quite well in Windows?

I'm only interested in cross platform stuff. Whichever part of the stack it is, it must work on Windows, Mac, and Linux.

Re: The genius and folly of MongoDB

#279

Post by sargun » Fri, Oct 25, 2013, 9:30 PM UTC

sitkack wrote: post #208

sargun wrote: post #136

Earlier quoted context omitted.

Riak also has massive problems. Realistically, figure out your data that you want to stick in a database, why, and how you're going to query it, and then work from there.

And those are what?

Your entire database's keyspace must fit in memory if you're using Bitcask. If you're using LevelDB, you have compaction overheads. Also, sibling resolution can get very messy, and complicated if your app works in a way that can potentially result in sibling explosions.

Re: The genius and folly of MongoDB

#280

Post by linuxhansl » Sat, Oct 26, 2013, 4:36 AM UTC

ddorian43 wrote: post #223

annnnd wrote: post #210

Earlier quoted context omitted.

I like your analogy, very fitting. To their defense however they are far from the only ones to do that... I lost about 2h worth of production data with HBase in just the same way - fortunately I didn't want to trust it completely anyway and had my own logs of all transactions on filesystem, but it definitely shattered my trust in that DB (not to mention it was a pain to setup and had no secondary indexes). I use Mong…

doesn't hbase persist in the transaction log every update, using append on hdfs ? or did you have a version of hdfs that didn't have append ?

HBase never needed nor used append. "Append" refers to the ability to reopen an existing file from a new client and append data to it. HBase writes only immutable files, which means they are written once then only read. Even the WAL is no exception; it is written once, and then only opened for read when needed for recovery or replication.

HBase needs hflush to make sure that the WAL edits are resident at at least 3 (default) HDFS data node machines.

Not sure how exactly grand parent lost data. Each edit is first written to the WAL then committed to the in memory store. The in memory store is flushed to disk into a new file at a certain size. If a server crashes and had unflushed data in the memory store that part of the data is replayed from the WAL on another server.