STM, CouchDB, and writing 5500 documents/second • Hacker News Board

Re: STM, CouchDB, and writing 5500 documents/second

#11

Post by technoweenie » Sat, Jul 24, 2010, 3:16 AM UTC

damienkatz wrote: post #10

saurik wrote: post #9

Earlier quoted context omitted.

The point of this article is queuing requestd at the webtier until you have enough to swamp the HTTP transport overhead to CouchDB. If the website crashes you are going to lose that data.

I'm not sure I understand. CouchDB won't lose any committed (fsynced) data when it crashes. And when configured, it won't return a success until it's committed. It's completely durable. edit: Oh I see, the stuff written in the clojure side won't be committed. If he wanted to write each one synchronously to CouchDB, that would work and CouchDB would still batch up the concurrent updates while writing.

"As requests come in, instead of connecting immediately to the database, why not queue them up until we have an optimal number and then do a bulk insert?"

Judging by the code, it looks like he stores about 50 documents before sending them to CouchDB. I believe the original poster was asking about the durability of Clojure STM, not Couch DB.

Re: STM, CouchDB, and writing 5500 documents/second

#12

Post by epochwolf » Sat, Jul 24, 2010, 3:19 AM UTC

damienkatz wrote: post #8

cloudhead wrote: post #7

...except it's not durable. If your server (clojure) crashes, you lose the data in memory.

It's quite durable. CouchDB has pure tail append storage and will fsync every update and batch concurrent updates into a single commit. It can get fairly close to a drives max sustained write speed.

If Clojure crashes, not the database.

Re: STM, CouchDB, and writing 5500 documents/second

#13

Post by apage43 » Sat, Jul 24, 2010, 4:38 AM UTC

cloudhead wrote: post #7

...except it's not durable. If your server (clojure) crashes, you lose the data in memory.

Except that's always the case. You mean if Clojure crashes before it queues up 50 docs and sends them, the docs in the queue are lost.

Durability is a concern when it means the integrity of your data is unknown. You'd probably be doing something like this in a bulk load, in which case you'd know where it crashed.

If this were queuing and batching inserts from many users it might be more of an issue (a user thinks something happened and it didn't), but realize that the at a queue size of 50 this performs at about ~5500 inserts/second. (And you can reduce this queue size if you don't need to insert that fast) This would affect users who happen to make their request within a roughly 0.18 millisecond window. If confirmation is still absolutely important, the post also links a version at the end that -does- wait for and return the IDs of the inserted documents from couchdb, guaranteeing the inserts have happened, and it still performs quite well.

Re: STM, CouchDB, and writing 5500 documents/second

#14

Post by rb2k_ » Sat, Jul 24, 2010, 8:35 AM UTC

jlcgull wrote: post #4

What about riak ? How does that compare ?

riak does support automatic sharding and enables you to tune your CAP paramters. Getting nodes in and out of a cluster is way easier with riak. While riak does also support map/reduce, generating something like a couchdb's "views" will probably end up being a problem since listing all the keys of large buckets (needed for a view of all your data) is a really expensive operation in riak. Also: couchdb supports incremental views, so if the view is fixed, you'll only need to update the changed documents. My main problem with couchDB was that in systems with a lot of changing data, the compaction was too slow for me. I actually changed data faster than the compaction could work on. Also: you have to compact that stuff yourself by sending an explicit http request.

Re: STM, CouchDB, and writing 5500 documents/second

#15

Post by swannodette » Sat, Jul 24, 2010, 1:44 PM UTC

cloudhead wrote: post #7

...except it's not durable. If your server (clojure) crashes, you lose the data in memory.

But not much more data than a webserver crash in a fully concurrent situation. One enhancement that I didn't bother to implement because it would muddy up the example was a timed flush so that data never just sits around in memory because the queue isn't full.

Re: STM, CouchDB, and writing 5500 documents/second

#16

Post by cpinto » Sat, Jul 24, 2010, 4:10 PM UTC

although interesting, bulk insert doesn't (didn't?) work across multiple databases managed by one host. and the ability to havve multiple databases is one of the things that makes couchdb a lot of fun.

Re: STM, CouchDB, and writing 5500 documents/second

#17

Post by papaf » Sat, Jul 24, 2010, 6:05 PM UTC

If the problem is the overhead of making HTTP connections, surely having a pool of open connections with keepalive solves the problem in a much more durable way? This is a setup I've seen in a telecom environment that supported a good throughput.

Re: STM, CouchDB, and writing 5500 documents/second

#18

Post by couchdb » Sat, Jul 24, 2010, 6:46 PM UTC

cpinto wrote: post #16

although interesting, bulk insert doesn't (didn't?) work across multiple databases managed by one host. and the ability to havve multiple databases is one of the things that makes couchdb a lot of fun.

the benefit of a bulk index is the ability to shove everything into a file and fsync once. so splitting across files won't really give the performance benefits.

Re: STM, CouchDB, and writing 5500 documents/second

#19

Post by mikealrogers » Thu, Jul 29, 2010, 2:42 PM UTC

jchrisa wrote: post #2

This is a follow up to the original post here http://dosync.posterous.com/22516635 This whole thing is making Clojure seem more interesting to me. I'd be curious to see how close Node.js can come to extracting this much write performance from CouchDB.

i find this whole thing a lot more of an experiment in clojure performance than CouchDB throughput.

he doesn't attach a handler to the response on his writes and is queuing writes in memory and bulking them. he's not concerned with durability at pretty much any layer.

this kind of setup is great for a large initial import of data but i would be weary of running it in production.

Re: STM, CouchDB, and writing 5500 documents/second

#20

Post by mikealrogers » Thu, Jul 29, 2010, 2:46 PM UTC

rb2k_ wrote: post #14

jlcgull wrote: post #4

What about riak ? How does that compare ?

riak does support automatic sharding and enables you to tune your CAP paramters. Getting nodes in and out of a cluster is way easier with riak. While riak does also support map/reduce, generating something like a couchdb's "views" will probably end up being a problem since listing all the keys of large buckets (needed for a view of all your data) is a really expensive operation in riak. Also: couchdb supports increme…

would you be willing to write up how your writes were outrunning the compactor in a little more detail in either a blog post or in an email to the apache couchdb dev list?

we've been talking about possible improvements to the compactor and having a solid use case that outruns it would help focus that.