Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On moving from CouchDB to Riak

labs.linkfluence.net

Post reply on HN
Print view View original

21–30 of 43 posts

Re: On moving from CouchDB to Riak

#21

Post by defroost »

philwise wrote: post #15

"We store a lot of data... 2TB" Given that a pair of 2TB drives is less than $250 on ebuyer right now, 2TB of data is not 'big data'. You could comfortably stuff that in any decent database (SQL Server for example, I'm sure PostgreSQL would work too). Just because a tiny machine on slicehost isn't big enough doesn't mean that your data won't fit in a normal database.

> You could comfortably stuff that in any decent database (SQL Server for example, I'm sure PostgreSQL would work too).

You may have noticed that the OP's requirements were: * a REST interface * sharding The costs of SQL Server are unbelievably high, and which you can do horizontal partitioning and sharding, Riak is designed to be used in this manner. On the other hand, re: SQL Server according to this article the "horizontal partitioning feature requires Enterprise or Datacenter licenses which have a retail price of 27,495 and 54,990 per processor." I think the OP made a wise choice. http://www.infoq.com/news/2011/02/SQL-Sharding

Re: On moving from CouchDB to Riak

#22

Post by rnewson »

biot wrote: post #18
rnewson wrote: post #11

Describing Bigcouch as a bit of hack while admitting you've not used it seems a little unfair. That said, it's very valuable to hear how other people see our product. Without doing a full-on sales pitch (I am not a salesman and do not portray one on TV), I should say that Bigcouch adds a lot of desired features to CouchDB, notably scaling beyond a single machine and sharding (which supports very large databases). We…

My reading was that they found CouchDB to have problems and adding complexity on top of a problematic base wasn't an appealing proposition for them.

My counterpoint is that Cloudant test and harden the CouchDB that we embed, it's not, for various reasons, a verbatim copy of CouchDB itself.

That said, vanilla CouchDB is quite stable. While there are always bugs, I don't recognize the system that the original poster is describing.

Re: On moving from CouchDB to Riak

#23

Post by JoachimSchipper »

defroost wrote: post #21
philwise wrote: post #15

"We store a lot of data... 2TB" Given that a pair of 2TB drives is less than $250 on ebuyer right now, 2TB of data is not 'big data'. You could comfortably stuff that in any decent database (SQL Server for example, I'm sure PostgreSQL would work too). Just because a tiny machine on slicehost isn't big enough doesn't mean that your data won't fit in a normal database.

> You could comfortably stuff that in any decent database (SQL Server for example, I'm sure PostgreSQL would work too). You may have noticed that the OP's requirements were: * a REST interface * sharding The costs of SQL Server are unbelievably high, and which you can do horizontal partitioning and sharding, Riak is designed to be used in this manner. On the other hand, re: SQL Server according to this article the "h…

You don't necessarily need to shard if the data fits comfortably on a standard disk... and the parent did mention PostgreSQL.

Re: On moving from CouchDB to Riak

#24

Post by grourk »

aphyr wrote: post #19
ot wrote: post #8

Earlier quoted context omitted.

> (and each key has, on the order of, 20 bits of overhead, IIRC) 20 bits ? Really? Less than an integer? Or did you mean bytes? (Not nitpicking here, I'm just curious)

I believe the limit using the current bitcask backend is (40 bytes + average key size) * (replication factor) / (number of cluster nodes * memory capacity of the smallest node). If that factor grows above 1, you can't store any more. IIRC correctly a 64-node cluster with 24gb of ram per node will handle a few billion 32-byte keys, replicated to three nodes. For larger keyspaces, the current recommendation is to use i…

Given that Riak seems to implement buckets "for free" by essentially making them key prefixes under-the-hood, does the bucket name size need to be considered as part of the key size? e.g., if my bucket name is 32 chars and my keys in that bucket are 32 chars, should I be using 64 bytes for average key size?

This is a question I've gotten conflicting answers to.

Re: On moving from CouchDB to Riak

#25

Post by davidhollander »

aphyr wrote: post #20
davidhollander wrote: post #13

What I disliked about Riak is that although at first glance it appears you can namespace key\values into multiple buckets you can't. The whole database is really only a single bucket and if you want to run a map\reduce it's always over every key in the database! Now you can simply increase complexity and create multiple Riak clusters for different data schemes and treat it as a replication tool. However, their in mem…

To my knowledge, there is no such thing as an in-memory bitcask backend for Riak. Could you have been using regular Bitcask (keeps values on disk in log-structured files) or the ETS memory backend?

Bitcask is an in memory datastore because it keeps a copy of the entire dataset in memory. 4GB of data==4GB of memory>trivial amount of memory used by a file descriptor.

>keeps values on disk in log-structured files

Of course it does. Log files are the easiest way to make an in memory datastructure persistant. Logs are generally used for periodically dropping fast writes as a backup in case you want to reload the memory datastructure in the future. Read requests are structured and served from the in memory datastructure and do not require a disk access.

The fact that you have something using all your RAM that should require 0 disk accesses for reads NOT substantially beating something performing thousands of disk accesses is a FUBAR situation.

Re: On moving from CouchDB to Riak

#26

Post by seiji »

davidhollander wrote: post #25
aphyr wrote: post #20

Earlier quoted context omitted.

To my knowledge, there is no such thing as an in-memory bitcask backend for Riak. Could you have been using regular Bitcask (keeps values on disk in log-structured files) or the ETS memory backend?

Bitcask is an in memory datastore because it keeps a copy of the entire dataset in memory. 4GB of data==4GB of memory>trivial amount of memory used by a file descriptor. > keeps values on disk in log-structured files Of course it does. Log files are the easiest way to make an in memory datastructure persistant. Logs are generally used for periodically dropping fast writes as a backup in case you want to reload the me…

Bitcask is an in memory datastore because it keeps a copy of the entire dataset in memo

Not quite. The keys are in memory along with disk position to retrieve the associated value with one seek.

Re: On moving from CouchDB to Riak

#27

Post by karmi »

> Each modification generates a new version of the document. (...) we don’t need it, and there’s no way to disable it (...)

Note that there's a `_revs_limit` setting available: " rel="nofollow">http://wiki.apache.org/couchdb/HTTP_database_API>. It's very beneficial for a use case like yours.

(Though I've seen CouchDB performing rather poorly when taking really heavy read/write load or compacting big datasets, on occasion.)

Re: On moving from CouchDB to Riak

#28

Post by Devilboy »

pkulak wrote: post #12
ot wrote: post #8

Earlier quoted context omitted.

> (and each key has, on the order of, 20 bits of overhead, IIRC) 20 bits ? Really? Less than an integer? Or did you mean bytes? (Not nitpicking here, I'm just curious)

20 bits can support a million records. As an average, per node or cluster, it seems reasonable to me.

Only a million records? That's waaaay too small.

Re: On moving from CouchDB to Riak

#29

Post by erikpukinskis »

philwise wrote: post #15

"We store a lot of data... 2TB" Given that a pair of 2TB drives is less than $250 on ebuyer right now, 2TB of data is not 'big data'. You could comfortably stuff that in any decent database (SQL Server for example, I'm sure PostgreSQL would work too). Just because a tiny machine on slicehost isn't big enough doesn't mean that your data won't fit in a normal database.

Just because your data will fit in a normal database doesn't mean it will be fast enough.

Re: On moving from CouchDB to Riak

#30

Post by SoftwareMaven »

ot wrote: post #8
smanek wrote: post #2

I went through a length evaluation process of Riak recently, and came away with a generally positive impression. First of all, it's beautifully engineered, as long as you just need a KV store or a graph DB (I wasn't in love with the MapReduce stuff, but that's another story). None of the hassle that Hadoop/Hbase have about some nodes being special (HBase Master, HDFS Namenode, etc). Also, no running multiple daemons…

> (and each key has, on the order of, 20 bits of overhead, IIRC) 20 bits ? Really? Less than an integer? Or did you mean bytes? (Not nitpicking here, I'm just curious)

I think 20 bits is too small. I've been looking for a source without success; the closest I've found[0] is 32-75 bytes per key which is something like 11 million keys per GB of RAM.

[0] http://www.quora.com/Matt-Heitzenroder

Post reply on HN