Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On moving from CouchDB to Riak

labs.linkfluence.net

Post reply on HN
Print view View original

11–20 of 43 posts

Re: On moving from CouchDB to Riak

#11

Post by rnewson »

Describing Bigcouch as a bit of hack while admitting you've not used it seems a little unfair. That said, it's very valuable to hear how other people see our product.

Without doing a full-on sales pitch (I am not a salesman and do not portray one on TV), I should say that Bigcouch adds a lot of desired features to CouchDB, notably scaling beyond a single machine and sharding (which supports very large databases). We run several clusters of Bigcouch nodes in production and have done for some time, it's a thrill for me personally to see a distributed database take a real pounding and keep on trucking.

I've been meaning to try Riak myself, so you've inspired me to finally pull down the code and give it a proper examination.

Re: On moving from CouchDB to Riak

#12

Post by pkulak »

ot wrote: post #8
smanek wrote: post #2

I went through a length evaluation process of Riak recently, and came away with a generally positive impression. First of all, it's beautifully engineered, as long as you just need a KV store or a graph DB (I wasn't in love with the MapReduce stuff, but that's another story). None of the hassle that Hadoop/Hbase have about some nodes being special (HBase Master, HDFS Namenode, etc). Also, no running multiple daemons…

> (and each key has, on the order of, 20 bits of overhead, IIRC) 20 bits ? Really? Less than an integer? Or did you mean bytes? (Not nitpicking here, I'm just curious)

20 bits can support a million records. As an average, per node or cluster, it seems reasonable to me.

Re: On moving from CouchDB to Riak

#13

Post by davidhollander »

What I disliked about Riak is that although at first glance it appears you can namespace key\values into multiple buckets you can't. The whole database is really only a single bucket and if you want to run a map\reduce it's always over every key in the database!

Now you can simply increase complexity and create multiple Riak clusters for different data schemes and treat it as a replication tool. However, their in memory bitcask map-reduce was actually slower than a hard-drive map-reduce on text JSON files in a folder. Where each file was individually opened, read from disk, encoded, and closed in a loop (with nothing held in memory). Which was rather scary!

Their replication scheme appears to be a pretty genuine copy of Amazon Dynamo though (consistent hashing ring, quorum, merkle trees) which is nice.

Re: On moving from CouchDB to Riak

#14

Post by electrum »

> We rolled out a cluster of 5 nodes and 1024 partitions. Each node has 8gig of memories, 3x1Tb disk in RAID5.

This hardware configuration doesn't make sense. 8 gigabytes of memory is pathetically small today. 32 gigs is basically standard, with 64 gigs costing little extra. RAID 5 with only three disks will have horrible performance. A single machine with 32/64 gigs and 12-16 disks in a better RAID configuration should perform better at a lower operational cost.

Re: On moving from CouchDB to Riak

#15

Post by philwise »

"We store a lot of data... 2TB"

Given that a pair of 2TB drives is less than $250 on ebuyer right now, 2TB of data is not 'big data'. You could comfortably stuff that in any decent database (SQL Server for example, I'm sure PostgreSQL would work too).

Just because a tiny machine on slicehost isn't big enough doesn't mean that your data won't fit in a normal database.

Re: On moving from CouchDB to Riak

#16

Post by rch »

davisp wrote: post #9
siculars wrote: post #6

Re. File size growth in Couch: Couch files are written in an append only fashion so that all operations are considered appends, such as updates. The main upside is durability meaning the file is always readable and never left in an odd state. As noted, however, this has the downside of requiring compaction to reclaim disk space. You should note that if you are using bitcask, the default backend for riak, you will hav…

The main two reasons that CouchDB doesn't use multiple files per database are system limits and increased complexity. CouchDB has a bit of an alternative design in that it accepts that people might be running a large number of databases on a single node. I don't remember the exact numbers but I think we've heard of deploys using 10-100K (small) db's on a single node. As to complexity, with a single file, there's no m…

I've noticed that interesting things tend to show up in bigcouch before couchdb. So, if I were starting a couchdb based project today, it would be with cloudant.

Re: On moving from CouchDB to Riak

#17

Post by [deleted] »

electrum wrote: post #14

> We rolled out a cluster of 5 nodes and 1024 partitions. Each node has 8gig of memories, 3x1Tb disk in RAID5. This hardware configuration doesn't make sense. 8 gigabytes of memory is pathetically small today. 32 gigs is basically standard, with 64 gigs costing little extra. RAID 5 with only three disks will have horrible performance. A single machine with 32/64 gigs and 12-16 disks in a better RAID configuration sho…

[deleted]

Re: On moving from CouchDB to Riak

#18

Post by biot »

rnewson wrote: post #11

Describing Bigcouch as a bit of hack while admitting you've not used it seems a little unfair. That said, it's very valuable to hear how other people see our product. Without doing a full-on sales pitch (I am not a salesman and do not portray one on TV), I should say that Bigcouch adds a lot of desired features to CouchDB, notably scaling beyond a single machine and sharding (which supports very large databases). We…

My reading was that they found CouchDB to have problems and adding complexity on top of a problematic base wasn't an appealing proposition for them.

Re: On moving from CouchDB to Riak

#19

Post by aphyr »

ot wrote: post #8
smanek wrote: post #2

I went through a length evaluation process of Riak recently, and came away with a generally positive impression. First of all, it's beautifully engineered, as long as you just need a KV store or a graph DB (I wasn't in love with the MapReduce stuff, but that's another story). None of the hassle that Hadoop/Hbase have about some nodes being special (HBase Master, HDFS Namenode, etc). Also, no running multiple daemons…

> (and each key has, on the order of, 20 bits of overhead, IIRC) 20 bits ? Really? Less than an integer? Or did you mean bytes? (Not nitpicking here, I'm just curious)

I believe the limit using the current bitcask backend is (40 bytes + average key size) * (replication factor) / (number of cluster nodes * memory capacity of the smallest node). If that factor grows above 1, you can't store any more.

IIRC correctly a 64-node cluster with 24gb of ram per node will handle a few billion 32-byte keys, replicated to three nodes. For larger keyspaces, the current recommendation is to use innodb, which doesn't need to keep keys in memory.

Re: On moving from CouchDB to Riak

#20

Post by aphyr »

davidhollander wrote: post #13

What I disliked about Riak is that although at first glance it appears you can namespace key\values into multiple buckets you can't. The whole database is really only a single bucket and if you want to run a map\reduce it's always over every key in the database! Now you can simply increase complexity and create multiple Riak clusters for different data schemes and treat it as a replication tool. However, their in mem…

To my knowledge, there is no such thing as an in-memory bitcask backend for Riak. Could you have been using regular Bitcask (keeps values on disk in log-structured files) or the ETS memory backend?
Post reply on HN