Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LevelDB: a fast and lightweight key/value database library

code.google.com

Post reply on HN
Print view View original

11–20 of 82 posts

Re: LevelDB: a fast and lightweight key/value database library

#11

Post by metabrew »

This is pretty neat, and different from many other "simple" KV stores, because it arranges things in key-order on disk, and permits forward/reverse iteration over keys. (ie, range queries are cheap)

edit: it's actually full log-structured merge, sstables, memtables etc.

Re: LevelDB: a fast and lightweight key/value database library

#12

Post by davidhollander »

ma2rten wrote: post #2

Does someone have any idea, how the benchmark results compare to other databases (e.g. redis) ?

Based on their numbers it looks slower than TokyoCabinet but still much faster than BerkleyDB. Would need to design a head to head benchmark.

I don't think Redis is a good comparison as that's an in-memory database so better suited for the 10% of hot data you'd need to cache. Whereas disk stores like TokyoCabinet and LevelDB would be great for storing the other 90-100%. If your use case involves a large dataset and you don't have terabytes of RAM lying around, that is.

Re: LevelDB: a fast and lightweight key/value database library

#14

Post by leif »

latch wrote: post #8
leif wrote: post #6

Anybody know what hardware those benchmarks were on? Is it actually durable? They mention problems with writeback caching, but are they at least durable from the OS perspective? How well does it scale?

it says so on the page: CPU: 4 x Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz Safe to assume the data fit into memory..whatever amount there was.

more interested in disk (and workloads larger than memory, but it doesn't sound like that's their use case)

Re: LevelDB: a fast and lightweight key/value database library

#15

Post by ivank »

cool-RR wrote: post #13

All the code files have this as a first line: // Copyright (c) 2011 The LevelDB Authors. All rights reserved. I wonder how come it's not copyrighted to Google.

That's what Google does when they set up a project that accepts contributions. Google Inc. is the author; see http://code.google.com/p/leveldb/source/browse/trunk/AUTHORS

Re: LevelDB: a fast and lightweight key/value database library

#18

Post by abarth »

cool-RR wrote: post #13

All the code files have this as a first line: // Copyright (c) 2011 The LevelDB Authors. All rights reserved. I wonder how come it's not copyrighted to Google.

The idea behind this approach is to simplify the maintenance of the copyright line and to encourage collective ownership of the code. I'm told that some open-source projects have conflicts about who's name appears in which files, but I've not experienced that first-hand.

Re: LevelDB: a fast and lightweight key/value database library

#19

Post by rb2k_ »

necubi wrote: post #10

Note that this is not a database server , like Redis or Memcached. This is a database library , more along the lines of sqlite. In particular: "There is no client-server support builtin to the library. An application that needs such support will have to wrap their own server around the library" and only one process can access a database a time.

It's the same model that Tokyo/Kyoto Cabinet uses. The difference is that Tokyo/Kyoto Tyrant is an available network interface :)

Re: LevelDB: a fast and lightweight key/value database library

#20

Post by sghemawat »

leif wrote: post #6

Anybody know what hardware those benchmarks were on? Is it actually durable? They mention problems with writeback caching, but are they at least durable from the OS perspective? How well does it scale?

It is durable from an OS perspective. I started typing in a long explanation, but I think the comment for the "sync" field in the options.h file captures things well:

  struct WriteOptions {
    // If true, the write will be flushed from the operating system
    // buffer cache (by calling WritableFile::Sync()) before the write
    // is considered complete.  If this flag is true, writes will be
    // slower.
    //
    // If this flag is false, and the machine crashes, some recent
    // writes may be lost.  Note that if it is just the process that
    // crashes (i.e., the machine does not reboot), no writes will be
    // lost even if sync==false.
    //
    // In other words, a DB write with sync==false has similar
    // crash semantics as the "write()" system call.  A DB write
    // with sync==true has similar crash semantics to a "write()"
    // system call followed by "fsync()".
    //
    // Default: false
    bool sync;
We don't have much experience with scaling to larger databases yet. There are known problems which will cause a (smallish) constant factor slowdown in write performance after the database becomes a few (10?) GB in size, but I don't recall the details all that well, and the implementation has changed somewhat since that experiment. I would like to characterize this better and fix things so we can support somewhere between 100GB-1TB databases well. It just hasn't become a priority yet.

The benchmark numbers on the linked page were from a small million entry database that easily fits in the OS buffer cache.

Post reply on HN