Benchmarking BDB, CDB and Tokyo Cabinet on large datasets • Hacker News Board

Re: Benchmarking BDB, CDB and Tokyo Cabinet on large datasets

#21

Post by jhy » Sat, May 23, 2009, 4:12 AM UTC

psadauskas wrote: post #4

vicaya wrote: post #3

Sorry, but 500MB DB size is a tiny dataset these days (anything < 1GB is tiny, < 4GB is small, < RAM on a single node (~8GB-64GB) is medium, < Disks on a single node (~128GB to a few TB) is large, huge dataset requires multiple nodes and typically above 128TBs.)

I agree, plus TC has a ton of parameters that can be tweaked, and the defaults are pretty small. The one that has the most pronouced effect is the bucket size, or the "width" of the hash table. The bigger, the less chance of collisions, which means you have to follow a linked-list to find the exact record. He used 11M keys, so a bnum in the range of 40M would be much quicker. I benchmarked TC b+tree on a 1TB db with…

Do you have a pointer to any more info on tuning TC? (Apart from the API guide)

Re: Benchmarking BDB, CDB and Tokyo Cabinet on large datasets

#22

Post by antirez » Sat, May 23, 2009, 10:32 AM UTC

It's a shame there is not Redis among the tested KV DBs. I guess the networking layer is not an option in this specific use case.

Re: Benchmarking BDB, CDB and Tokyo Cabinet on large datasets

#23

Post by kree10 » Sat, May 23, 2009, 11:27 AM UTC

asb wrote: post #11

It would be nice if DJB would release CDB under the public domain like he has now done with most of his other software. However, there's a great public domain implementation by Michael Tokarev I'd recommend: http://www.corpit.ru/mjt/tinycdb.html

The first lines of cdb.c and cdb.h read "/* Public domain */". I would expect DJB to spell this out more explicitly if he releases an updated cdb package. I say "if" because the last release was Feb 2000.

Re: Benchmarking BDB, CDB and Tokyo Cabinet on large datasets

#24

Post by asb » Sat, May 23, 2009, 11:45 AM UTC

kree10 wrote: post #23

asb wrote: post #11

It would be nice if DJB would release CDB under the public domain like he has now done with most of his other software. However, there's a great public domain implementation by Michael Tokarev I'd recommend: http://www.corpit.ru/mjt/tinycdb.html

The first lines of cdb.c and cdb.h read "/* Public domain */". I would expect DJB to spell this out more explicitly if he releases an updated cdb package. I say "if" because the last release was Feb 2000.

I hadn't noticed that. Wikipedia explains the licensing in more details (http://en.wikipedia.org/wiki/Constant_Data_Base#Library) - the library code is public domain (and included in the public domain djbdns), but the rest of the package is under djb's annoying "You may distribute unmodified copies" license.