Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RocksDB – A persistent key-value store for fast storage environments

rocksdb.org

Post reply on HN
Print view View original

11–20 of 75 posts

Re: RocksDB – A persistent key-value store for fast storage environments

#11

Post by dhruba_b »

gfodor wrote: post #2

this is cool, though I'd wonder how it compares to Kyoto Cabinet. another big issue I've run into personally is the fact that both LevelDB and KC don't explicitly support multiple processes reading the db at once. (KC's API allows this but advises against it, LevelDB afaik doesn't even allow it.) I wonder if RocksDB gets past this.

RocksDB allows only one process to open the DB in read-write mode but other process can access the DB read-only (with a few configuration settings)

Re: RocksDB – A persistent key-value store for fast storage environments

#12

Post by mml »

dhruba_b wrote: post #10

Hi guys, I am Dhruba and I work in the Database Engineering team at Facebook. We just released RocksDB as an open source project. If anybody has any technical questions about RocksDB, please feel free to ask. Thanks.

Any replication support? (Or any sort of distribution?)

Re: RocksDB – A persistent key-value store for fast storage environments

#14

Post by justin66 »

AaronFriel wrote: post #5
rdtsc wrote: post #3

Well LevelDB is already good. And if this improves on it, that's great. I was looking at embedded key value stores and also found -- HyperLevelDB (from creators of Hyperdex database). They also improved on LevelDB in respect to compaction and locking: http://hyperdex.org/performance/leveldb/ So now I am curios how it would compare. Another interesting case optimized for reads is LMDB. That is a small but very fast em…

The LMDB statistics are very strange - why is synchronous SSD performance worse on most figures than HDD performance? Something seems very wrong with these benchmarks: Section 5 (SSD) F (Synchronous Writes) Random Writes LevelDB 342 ops/sec Kyoto TreeDB 67 ops/sec SQLite3 114 ops/sec MDB 148 ops/sec MDB, no MetaSync 322 ops/sec BerkeleyDB 291 ops/sec Section 8 (HDD) F (Synchronous Writes) Random Writes LevelDB 1291 o…

Paging hyc_symas. Howard Chu isn't shy about talking about benchmark results and he can be found here and on twitter.

Re: RocksDB – A persistent key-value store for fast storage environments

#16

Post by techtalsky »

dhruba_b wrote: post #10

Hi guys, I am Dhruba and I work in the Database Engineering team at Facebook. We just released RocksDB as an open source project. If anybody has any technical questions about RocksDB, please feel free to ask. Thanks.

Can this be used as a drop-in replacement for LevelDB on queue technologies like ApolloMQ that use LevelDB as the default?

Re: RocksDB – A persistent key-value store for fast storage environments

#17

Post by jbapple »

dhruba_b wrote: post #10

Hi guys, I am Dhruba and I work in the Database Engineering team at Facebook. We just released RocksDB as an open source project. If anybody has any technical questions about RocksDB, please feel free to ask. Thanks.

Hi Dhruba, thanks for volunteering to ask questions.

What are the big algorithmic ideas behind RocksDB?

My understanding is that LevelDB is based on log structured merge trees. These can be deamortized using methods from Overmars's "The Design of Dynamic Data Structures" or Bender et al.'s "Cache-Oblivious Streaming B-trees". How did you reduce latency?

What else was slowing down databases larger than RAM? How did you fix that?

Re: RocksDB – A persistent key-value store for fast storage environments

#18

Post by jonstewart »

mml wrote: post #12
dhruba_b wrote: post #10

Hi guys, I am Dhruba and I work in the Database Engineering team at Facebook. We just released RocksDB as an open source project. If anybody has any technical questions about RocksDB, please feel free to ask. Thanks.

Any replication support? (Or any sort of distribution?)

The primary enhancements over LevelDB seem to be parallel compactions of disjoint ranges, to take advantage of cheap seeks on flash storage, and the ability to parameterize core algorithms and data structures to suit a particular anticipated workload. All very cool; anything else major?

Also, there aren't JNI bindings... are there?

Thanks for the contribution. Just started using LevelDB on a project, but deployment will involve fast flash storage and rocksdb looks like a worthy successor.

Jon

Re: RocksDB – A persistent key-value store for fast storage environments

#19

Post by hendzen »

dhruba_b wrote: post #10

Hi guys, I am Dhruba and I work in the Database Engineering team at Facebook. We just released RocksDB as an open source project. If anybody has any technical questions about RocksDB, please feel free to ask. Thanks.

How much do you think RocksDB/LevelDB performance is impacted by the use of relatively coarse-grained locking? Another LevelDB fork, HyperLevelDB [1] implemented a fine grained scheme with performance benefits. Disclaimer: I am working on a (unreleased) fork of LevelDB that uses hardware transactional memory for synchronization using the new TSX-NI instructions present on Haswell processors.

[1] - http://hyperdex.org/performance/leveldb/

Re: RocksDB – A persistent key-value store for fast storage environments

#20

Post by droid5 »

dhruba_b wrote: post #10

Hi guys, I am Dhruba and I work in the Database Engineering team at Facebook. We just released RocksDB as an open source project. If anybody has any technical questions about RocksDB, please feel free to ask. Thanks.

https://github.com/facebook/rocksdb/issues/new is a 404.

Appreciated when project is on github and open for issues/PR, etc.

Post reply on HN