Benchmarking BDB, CDB and Tokyo Cabinet on large datasets
1–10 of 24 posts
Re: Benchmarking BDB, CDB and Tokyo Cabinet on large datasets
#2I'm a bit surprised at the poor showing of Tokyo Cabinet in this test, given the amount of buzz it's gathered in the last few months, but the CDB results honestly don't surprise me in the least -- it's a special-purpose datastructure wrapped in a thin library that just smokes any other DB type if your workload allows for a complete rebuild of the database on each update.
Re: Benchmarking BDB, CDB and Tokyo Cabinet on large datasets
#3Sorry, but 500MB DB size is a tiny dataset these days (anything < 1GB is tiny, < 4GB is small, < RAM on a single node (~8GB-64GB) is medium, < Disks on a single node (~128GB to a few TB) is large, huge dataset requires multiple nodes and typically above 128TBs.)
Re: Benchmarking BDB, CDB and Tokyo Cabinet on large datasets
#4Sorry, but 500MB DB size is a tiny dataset these days (anything < 1GB is tiny, < 4GB is small, < RAM on a single node (~8GB-64GB) is medium, < Disks on a single node (~128GB to a few TB) is large, huge dataset requires multiple nodes and typically above 128TBs.)
I agree, plus TC has a ton of parameters that can be tweaked, and the defaults are pretty small. The one that has the most pronouced effect is the bucket size, or the "width" of the hash table. The bigger, the less chance of collisions, which means you have to follow a linked-list to find the exact record. He used 11M keys, so a bnum in the range of 40M would be much quicker.
I benchmarked TC b+tree on a 1TB db with ~350M keys, and it worked great. I would publish the numbers, but I'm embarrassed that they aren't very rigorous.
cdb docs say it has a limit of 4GB, which makes it pretty much worthless for anything I would use it for.
Re: Benchmarking BDB, CDB and Tokyo Cabinet on large datasets
#5In my experience, the downside of BDB is simply the license. There is no way for me to use it in my program (and distribute my program) without opening up my source. I can't simply 'query' a BDB server - by using BDB at all I have to link to it and also open source my own code as part of the Sleepycat License (or so I've been told - IANAL).
Re: Benchmarking BDB, CDB and Tokyo Cabinet on large datasets
#6Sorry, but 500MB DB size is a tiny dataset these days (anything < 1GB is tiny, < 4GB is small, < RAM on a single node (~8GB-64GB) is medium, < Disks on a single node (~128GB to a few TB) is large, huge dataset requires multiple nodes and typically above 128TBs.)
I agree, plus TC has a ton of parameters that can be tweaked, and the defaults are pretty small. The one that has the most pronouced effect is the bucket size, or the "width" of the hash table. The bigger, the less chance of collisions, which means you have to follow a linked-list to find the exact record. He used 11M keys, so a bnum in the range of 40M would be much quicker. I benchmarked TC b+tree on a 1TB db with…
I guess someone could tweak cdb to be >4GB, but these C libraries are always insanely convoluted.
Re: Benchmarking BDB, CDB and Tokyo Cabinet on large datasets
#7In my experience, the downside of BDB is simply the license. There is no way for me to use it in my program (and distribute my program) without opening up my source. I can't simply 'query' a BDB server - by using BDB at all I have to link to it and also open source my own code as part of the Sleepycat License (or so I've been told - IANAL).
DB up to and including 1.86 are Berkeley license (basically 'do anything with this but include this notice' but without open source requirements), but DB up to and including 1.86 only allow I tried to upgrade it with 64-bit integers, but it's such convoluted C code I can't make head nor tail of it.
Also try buying BDB ($20,000). I emailed them asking for a discount, but Oracle (who owns them now) never replied. :-(
Re: Benchmarking BDB, CDB and Tokyo Cabinet on large datasets
#8Sorry, but 500MB DB size is a tiny dataset these days (anything < 1GB is tiny, < 4GB is small, < RAM on a single node (~8GB-64GB) is medium, < Disks on a single node (~128GB to a few TB) is large, huge dataset requires multiple nodes and typically above 128TBs.)
I agree, plus TC has a ton of parameters that can be tweaked, and the defaults are pretty small. The one that has the most pronouced effect is the bucket size, or the "width" of the hash table. The bigger, the less chance of collisions, which means you have to follow a linked-list to find the exact record. He used 11M keys, so a bnum in the range of 40M would be much quicker. I benchmarked TC b+tree on a 1TB db with…
Actually, the core CDB data structure only limits keys and values to 4GB each, not in total:
http://www.unixuser.org/~euske/doc/cdbinternals/index.html
The hash algorithm used also only produces a 32-bit key, meaning you'll be limited to 2^32 total records. Again, though, unless your data is of trivial size, that gives you considerably more room to work with than a hard 4GB limit.
Edit: doh! can't use double-asterisk for exponent on HN
Re: Benchmarking BDB, CDB and Tokyo Cabinet on large datasets
#9I'm a bit surprised at the poor showing of Tokyo Cabinet in this test, given the amount of buzz it's gathered in the last few months, but the CDB results honestly don't surprise me in the least -- it's a special-purpose datastructure wrapped in a thin library that just smokes any other DB type if your workload allows for a complete rebuild of the database on each update.
I am a bit confused by the table. "bdb_btree 14.3/s " means 14.3 records per second? Is that the correct interpretation?
Re: Benchmarking BDB, CDB and Tokyo Cabinet on large datasets
#10Earlier quoted context omitted.
I agree, plus TC has a ton of parameters that can be tweaked, and the defaults are pretty small. The one that has the most pronouced effect is the bucket size, or the "width" of the hash table. The bigger, the less chance of collisions, which means you have to follow a linked-list to find the exact record. He used 11M keys, so a bnum in the range of 40M would be much quicker. I benchmarked TC b+tree on a 1TB db with…
Actually, the core CDB data structure only limits keys and values to 4GB each , not in total: http://www.unixuser.org/~euske/doc/cdbinternals/index.html The hash algorithm used also only produces a 32-bit key, meaning you'll be limited to 2^32 total records. Again, though, unless your data is of trivial size, that gives you considerably more room to work with than a hard 4GB limit. Edit: doh! can't use double-asteris…
[deleted]