LevelDB: A Fast Persistent Key-Value Store • Hacker News Board

Re: LevelDB: A Fast Persistent Key-Value Store

#41

Post by agazso » Wed, Jul 27, 2011, 9:52 PM UTC

gleb wrote: post #33

The synchronous writes benchmark is interesting. This is normally bound by # seeks your disk can do per second, which is mostly a function of rotational speed. With 7200RPM drive you get 7200/60 = 120 of these a second. So the 100 and 110 numbers for competitors make sense. 2,400 for LevelDB does not. Is LevelDB batching writes or is there something more interesting going on?

If you are writing sequentially, then you can write more than the number of seeks.

And that is exactly what LevelDB is doing: writing a log (sequential), and when the memorychunk is full, it is writing it to disk sorted (this is also sequential).

Re: LevelDB: A Fast Persistent Key-Value Store

#42

Post by sallen » Wed, Jul 27, 2011, 10:03 PM UTC

gaborcselle wrote: post #9

Hi there! I'm a YC alum (reMail W09) and helped Jeff and Sanjay with LevelDB. Let me know if you have any questions about LevelDB and I'll see if I can help.

Does this system have transactions and ACID guarantees?

Re: LevelDB: A Fast Persistent Key-Value Store

#43

Post by davidhollander » Wed, Jul 27, 2011, 10:05 PM UTC

stephth wrote: post #36

davidhollander wrote: post #35

Earlier quoted context omitted.

BDB is a key\value store for unordered data more similar to Tokyo Cabinet hash databases. Tokyo Cabinet hash databases are a much faster option than BDB if you only need unordered data. LevelDB is for if you need ordered data, and a more appropriate comparison would be against a B+\tree database.

LevelDB is for if you need ordered data LevelDB is slower with random reads, but that doesn't mean you shouldn't use it for unordered data - it's still quite fast.

>LevelDB is slower with random reads, but that doesn't mean you shouldn't use it for unordered data - it's still quite fast.

In a positive analysis (should rather than shouldn't), assuming no default choice, it seems rational to use Tokyo Cabinet or CDB _hashmaps_ for unordered data, and LevelDB for ordered data, from a datastructure and performance standpoint. To assert more would probably need a specific use case for context.

Re: LevelDB: A Fast Persistent Key-Value Store

#44

Post by gaborcselle » Wed, Jul 27, 2011, 10:14 PM UTC

gojomo wrote: post #28

An interesting development a while back that I'm surprised hasn't received more attention was Oracle's release of a SQLite-based interface to BDB: http://www.oracle.com/technetwork/database/berkeleydb/overvi... It's essentially drop-in compatible with SQLite, but with added concurrency and speed for most operations. (The concurrency addresses a major issue usually keeping SQLite as a prototyping/single-user-only opti…

Thanks for the link! You could theoretically just compile this file against SQLite-based BDB: http://code.google.com/p/leveldb/source/browse/trunk/doc/ben... And get the numbers yourself. (If you do, please post them here.)

Re: LevelDB: A Fast Persistent Key-Value Store

#45

Post by mumrah » Wed, Jul 27, 2011, 10:15 PM UTC

Anyone know how LevelDB compares to Voldemort? From a cursory glance, they are identical in their simple API (get, put, delete)

Re: LevelDB: A Fast Persistent Key-Value Store

#46

Post by gaborcselle » Wed, Jul 27, 2011, 10:17 PM UTC

swah wrote: post #19

Interesting how, like in the open-sourced protobuf, there are no commits by Jeff or Sanjay...

dgrogan and myself have been batching changes to LevelDB from our internal code repository to put them on the Google Code page. Playing Google Code site admin didn't seem to me like a good use of Jeff and Sanjay's time.

Re: LevelDB: A Fast Persistent Key-Value Store

#47

Post by stephth » Wed, Jul 27, 2011, 10:49 PM UTC

davidhollander wrote: post #43

stephth wrote: post #36

Earlier quoted context omitted.

LevelDB is for if you need ordered data LevelDB is slower with random reads, but that doesn't mean you shouldn't use it for unordered data - it's still quite fast.

> LevelDB is slower with random reads, but that doesn't mean you shouldn't use it for unordered data - it's still quite fast. In a positive analysis (should rather than shouldn't), assuming no default choice, it seems rational to use Tokyo Cabinet or CDB _hashmaps_ for unordered data, and LevelDB for ordered data, from a datastructure and performance standpoint. To assert more would probably need a specific use case…

In a positive analysis (should rather than shouldn't), assuming no default choice, it seems rational to use Tokyo Cabinet or CDB _hashmaps_ for unordered data, and LevelDB for ordered data, from a datastructure and performance standpoint.

It's as rational as doing optimizations. If the specific performance is extremely critical, yes, it definitely makes sense. But LevelDB does well with random reads [1]. With all its features and its permissive license, I think it's a strong contender as developer's go-to embedded key-value db, like SQLite for relational data.

Don't get me wrong, I do think your comparison is valuable, but I'm afraid it could be misleading; I specially found the wording LevelDB is for if you need ordered data misleading. Someone could read it and assume that LevelDB doesn't do well with unordered data.

[1] Compare today's benchmarks with these http://fallabs.com/tokyocabinet/benchmark.pdf, it looks like random reads in LevelDB are quite faster than BDB.

Re: LevelDB: A Fast Persistent Key-Value Store

#48

Post by gaborcselle » Wed, Jul 27, 2011, 11:02 PM UTC

mumrah wrote: post #45

Anyone know how LevelDB compares to Voldemort? From a cursory glance, they are identical in their simple API (get, put, delete)

My understanding is that Voldemort is a distributed key-value storage system, while LevelDB is a local on-disk key-value storage system.

Re: LevelDB: A Fast Persistent Key-Value Store

#49

Post by swah » Wed, Jul 27, 2011, 11:52 PM UTC

gaborcselle wrote: post #46

swah wrote: post #19

Interesting how, like in the open-sourced protobuf, there are no commits by Jeff or Sanjay...

dgrogan and myself have been batching changes to LevelDB from our internal code repository to put them on the Google Code page. Playing Google Code site admin didn't seem to me like a good use of Jeff and Sanjay's time.

Yep, its great you guys could separate it from internal dependencies! Congrats.

Re: LevelDB: A Fast Persistent Key-Value Store

#50

Post by groby_b » Thu, Jul 28, 2011, 12:01 AM UTC

gleb wrote: post #40

groby_b wrote: post #37

Earlier quoted context omitted.

If you write full disk blocks, wouldn't the disk cache hide the seek latency?

Having write disk cache on would certainly explain it. But that leaves the question of discrepancy with numbers with competitors. You turn off write-through caching on disks when you run a database unless you are willing to accept corruption (which is worse than data loss) on power outage. And that's why you can't get acceptable write performance out of database without a battery-backed RAID controller (or something…

If cache is on, any performance discrepancy can be explained away by "usage patterns" :)

Also, do you really mean turn off write-through, or did you mean write-behind? (I can't see how write-through would cause corruption, but maybe I'm missing something...)

Also, I wouldn't be surprised if there's a discrepancy in the flushing code across systems. God knows flushing a file to disk in cross-platform code is an arcane science :)

And finally, as somebody else pointed out, LevelDB seems to order write access sequentially as much as possible.