STM, CouchDB, and writing 5500 documents/second
dosync.posterous.com
STM, CouchDB, and writing 5500 documents/second
1–10 of 21 posts
Re: STM, CouchDB, and writing 5500 documents/second
#2This is a follow up to the original post here http://dosync.posterous.com/22516635
This whole thing is making Clojure seem more interesting to me. I'd be curious to see how close Node.js can come to extracting this much write performance from CouchDB.
Re: STM, CouchDB, and writing 5500 documents/second
#3This is a follow up to the original post here http://dosync.posterous.com/22516635 This whole thing is making Clojure seem more interesting to me. I'd be curious to see how close Node.js can come to extracting this much write performance from CouchDB.
For me, the problem has always been the indexed views taking too long to generate for a large amount of documents on first view. I know that some people do a bulk update, then hit the different indexed views to get them updated incrementally.
What can be done to increase the speed of the first view of new indexed views that you might have later on? Do you just have to wait for it?
What about ad-hoc views? Are there ways to speed that up?
Re: STM, CouchDB, and writing 5500 documents/second
#4What about riak ? How does that compare ?
Re: STM, CouchDB, and writing 5500 documents/second
#5This is a follow up to the original post here http://dosync.posterous.com/22516635 This whole thing is making Clojure seem more interesting to me. I'd be curious to see how close Node.js can come to extracting this much write performance from CouchDB.
For me, the problem has always been the indexed views taking too long to generate for a large amount of documents on first view. I know that some people do a bulk update, then hit the different indexed views to get them updated incrementally. What can be done to increase the speed of the first view of new indexed views that you might have later on? Do you just have to wait for it? What about ad-hoc views? Are there w…
The key is: don't experiment / develop on a giant database. Make a small subset of the database for testing views on. Then they are plenty fast.
As far as generation speed, what matters is that the indexer can keep up with the insert rate. Unless you are doing a big import from an existing dataset, you'll have to have A LOT of user activity to generate so much data that you are outrunning the indexer. In that case you probably have a big enough project that it makes sense to use a cluster solution like CouchDB-Lounge (which will divide the generation time by roughly the # of hosts you bring into the cluster).
Someday soon I hope we'll have an integrated clustering solution (Cloudant is working to contribute theirs back to the project) so you can just deploy to a set of nodes and get these benefits without much additional operational complexity.
Re: STM, CouchDB, and writing 5500 documents/second
#6[deleted]
Re: STM, CouchDB, and writing 5500 documents/second
#7...except it's not durable. If your server (clojure) crashes, you lose the data in memory.
Re: STM, CouchDB, and writing 5500 documents/second
#8...except it's not durable. If your server (clojure) crashes, you lose the data in memory.
It's quite durable. CouchDB has pure tail append storage and will fsync every update and batch concurrent updates into a single commit. It can get fairly close to a drives max sustained write speed.
Re: STM, CouchDB, and writing 5500 documents/second
#9...except it's not durable. If your server (clojure) crashes, you lose the data in memory.
It's quite durable. CouchDB has pure tail append storage and will fsync every update and batch concurrent updates into a single commit. It can get fairly close to a drives max sustained write speed.
The point of this article is queuing requestd at the webtier until you have enough to swamp the HTTP transport overhead to CouchDB. If the website crashes you are going to lose that data.
Re: STM, CouchDB, and writing 5500 documents/second
#10Earlier quoted context omitted.
It's quite durable. CouchDB has pure tail append storage and will fsync every update and batch concurrent updates into a single commit. It can get fairly close to a drives max sustained write speed.
The point of this article is queuing requestd at the webtier until you have enough to swamp the HTTP transport overhead to CouchDB. If the website crashes you are going to lose that data.
I'm not sure I understand. CouchDB won't lose any committed (fsynced) data when it crashes. And when configured, it won't return a success until it's committed. It's completely durable.
edit: Oh I see, the stuff written in the clojure side won't be committed. If he wanted to write each one synchronously to CouchDB, that would work and CouchDB would still batch up the concurrent updates while writing.