Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LevelDB: A Fast Persistent Key-Value Store

google-opensource.blogspot.com

Post reply on HN
Print view View original

61–66 of 66 posts

Re: LevelDB: A Fast Persistent Key-Value Store

#61

Post by leif »

gleb wrote: post #33

The synchronous writes benchmark is interesting. This is normally bound by # seeks your disk can do per second, which is mostly a function of rotational speed. With 7200RPM drive you get 7200/60 = 120 of these a second. So the 100 and 110 numbers for competitors make sense. 2,400 for LevelDB does not. Is LevelDB batching writes or is there something more interesting going on?

Data structures which require a disk seek per random insert are obsolete. LevelDB is using a Log-Structured Merge Tree, one of many write-optimized data structures (but not the best).

Re: LevelDB: A Fast Persistent Key-Value Store

#62

Post by leif »

agazso wrote: post #41
gleb wrote: post #33

The synchronous writes benchmark is interesting. This is normally bound by # seeks your disk can do per second, which is mostly a function of rotational speed. With 7200RPM drive you get 7200/60 = 120 of these a second. So the 100 and 110 numbers for competitors make sense. 2,400 for LevelDB does not. Is LevelDB batching writes or is there something more interesting going on?

If you are writing sequentially, then you can write more than the number of seeks. And that is exactly what LevelDB is doing: writing a log (sequential), and when the memorychunk is full, it is writing it to disk sorted (this is also sequential).

flushing the log in an LSM is only kinda sequential, sadly

Re: LevelDB: A Fast Persistent Key-Value Store

#63

Post by DanWaterworth »

leif wrote: post #61
gleb wrote: post #33

The synchronous writes benchmark is interesting. This is normally bound by # seeks your disk can do per second, which is mostly a function of rotational speed. With 7200RPM drive you get 7200/60 = 120 of these a second. So the 100 and 110 numbers for competitors make sense. 2,400 for LevelDB does not. Is LevelDB batching writes or is there something more interesting going on?

Data structures which require a disk seek per random insert are obsolete. LevelDB is using a Log-Structured Merge Tree, one of many write-optimized data structures (but not the best).

This link, comparing LSM trees with fractal trees, is quite interesting: http://www.quora.com/What-are-the-major-differences-between-...

Re: LevelDB: A Fast Persistent Key-Value Store

#64

Post by pjscott »

dlsspy wrote: post #58
aashay wrote: post #51

Earlier quoted context omitted.

How does this compare to other persistent key-value stores such as Membase?

Membase is a clustered data storage service your application uses. LevelDB is a persistence library. That makes LevelDB the kind of thing you plug into membase to get the unique properties it has to offer (or at least for fun).

In fact, the Riak guys are planning on doing exactly that: offering LevelDB as one of the storage back-ends, perhaps even the default.

http://blog.basho.com/2011/07/01/Leveling-the-Field/

Re: LevelDB: A Fast Persistent Key-Value Store

#66

Post by gojomo »

est wrote: post #57
gojomo wrote: post #28

An interesting development a while back that I'm surprised hasn't received more attention was Oracle's release of a SQLite-based interface to BDB: http://www.oracle.com/technetwork/database/berkeleydb/overvi... It's essentially drop-in compatible with SQLite, but with added concurrency and speed for most operations. (The concurrency addresses a major issue usually keeping SQLite as a prototyping/single-user-only opti…

> SQLite-based interface to BDB One thing I didn't get about SQL API for BDB, how does something like select * from users where name!='tom' work ?

You really don't know or care that you're using BDB; it works (to the user) just like SQLite. (Behind the scenes, it's using BDB for the tables/indexes, and so would do various full- or partial- table scans much like SQLite's native on-disk format.)
Post reply on HN