Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RocksDB – A persistent key-value store for fast storage environments

rocksdb.org

Post reply on HN
Print view View original

41–50 of 75 posts

Re: RocksDB – A persistent key-value store for fast storage environments

#41

Post by dhruba_b »

jonstewart wrote: post #18
mml wrote: post #12

Earlier quoted context omitted.

Any replication support? (Or any sort of distribution?)

The primary enhancements over LevelDB seem to be parallel compactions of disjoint ranges, to take advantage of cheap seeks on flash storage, and the ability to parameterize core algorithms and data structures to suit a particular anticipated workload. All very cool; anything else major? Also, there aren't JNI bindings... are there? Thanks for the contribution. Just started using LevelDB on a project, but deployment w…

Thanks for your comments Jon. RocksDB shares some of its genes with LevelDB.. something like a parent-child relationship.

Please check out Universal Comaction Style, multi-threaded-compaction, pipelined memtables.

I used to have JNI bindings that I pulled in from https://github.com/fusesource/leveldbjni but it was difficult for me to update the JNI everytime we added new apis to RocksDB. It would be great if somebody who needs Java support can implement JNI bindings for RocksDB.

Re: RocksDB – A persistent key-value store for fast storage environments

#42

Post by dhruba_b »

droid5 wrote: post #20
dhruba_b wrote: post #10

Hi guys, I am Dhruba and I work in the Database Engineering team at Facebook. We just released RocksDB as an open source project. If anybody has any technical questions about RocksDB, please feel free to ask. Thanks.

https://github.com/facebook/rocksdb/issues/new is a 404. Appreciated when project is on github and open for issues/PR, etc.

are you not able to access the github repo?

Re: RocksDB – A persistent key-value store for fast storage environments

#43

Post by ithkuil »

jbellis wrote: post #32
hyc_symas wrote: post #24

Earlier quoted context omitted.

Keep in mind, the HDD was using ext2 and the SSD was using reiserfs. Synchronous writes on ext2 are faster than all journaling filesystems.

Not three orders of magnitude faster, which is the difference between hdd and ssd random writes.

Three orders of magnitude faster would mean 1000x faster. You probably meant 3 times faster.

Re: RocksDB – A persistent key-value store for fast storage environments

#44

Post by jbapple »

dhruba_b wrote: post #35
jbapple wrote: post #17

Earlier quoted context omitted.

Hi Dhruba, thanks for volunteering to ask questions. What are the big algorithmic ideas behind RocksDB? My understanding is that LevelDB is based on log structured merge trees. These can be deamortized using methods from Overmars's "The Design of Dynamic Data Structures" or Bender et al.'s "Cache-Oblivious Streaming B-trees". How did you reduce latency? What else was slowing down databases larger than RAM? How did yo…

RocksDB has an LSM architecture, similar in nature to HBase, leveldb, etc. But the implementation is based on a Theorem that we will be publishing shortly. I am working on the Theorem with a colleague of mine. Cache Oblivious B-trees is an interesting paper. Similarly fractal trees. Most of them optimize the case when index nodes are not in memory. However, in our use-cases, we typically configure the system in such…

Can you share with us the statement of that theorem?

What is "UniversalStyleCompaction", and why is it capitalized and missing spaces?

How does a Bloom filter for range scans work? Standard Bloom filters (as you know) are for existence only.

Re: RocksDB – A persistent key-value store for fast storage environments

#45

Post by droid5 »

dhruba_b wrote: post #42
droid5 wrote: post #20

Earlier quoted context omitted.

https://github.com/facebook/rocksdb/issues/new is a 404. Appreciated when project is on github and open for issues/PR, etc.

are you not able to access the github repo?

doesn't compile on centos 6, gcc 4.6.3 wanted to file a issue. but 404 stopped me.

Re: RocksDB – A persistent key-value store for fast storage environments

#46

Post by vicaya »

hyc_symas wrote: post #27
rdtsc wrote: post #3

Well LevelDB is already good. And if this improves on it, that's great. I was looking at embedded key value stores and also found -- HyperLevelDB (from creators of Hyperdex database). They also improved on LevelDB in respect to compaction and locking: http://hyperdex.org/performance/leveldb/ So now I am curios how it would compare. Another interesting case optimized for reads is LMDB. That is a small but very fast em…

LSMs have a long long way to go to catch up to LMDB. http://symas.com/mdb/hyperdex/

For reads, sure. LSMs are optimized for writes, while LMDB, which is a nice B-tree implementation is optimized for reads.

LSMs are getting popular because it's harder to scale durable writes than reads, which can be handled (in many cases independently) by caching.

Re: RocksDB – A persistent key-value store for fast storage environments

#48

Post by Patient0 »

I'm surprised that the C++ code is not using the RAII idiom in some obvious places.

For example: https://github.com/facebook/rocksdb/blob/master/db/db_impl.c

There are many places with bracketed calls to mutex_.Lock and mutex_.Unlock().

An example:

      mutex_.Unlock();
      LogFlush(options_.info_log);
      env_->SleepForMicroseconds(1000000);
      mutex_.Lock()
Why didn't the authors use the RAII idiom here? Even if there are no exceptions expected, the code would still be simpler and less error prone by using a guard object.

Re: RocksDB – A persistent key-value store for fast storage environments

#49

Post by kitsune_ »

hyc_symas wrote: post #31
rdtsc wrote: post #30

Earlier quoted context omitted.

Makes sense. Most impressive about LMDB to me is the zero-copy model for readers, with is no extra memcpy needed, maybe that is something obvious for database gurus but it is pretty clever trick I think.

It's pretty significant, yes. Eliminating multiple copies of everything got us a 4:1 reduction in memory footprint in OpenLDAP slapd (compared to our BerkeleyDB-based backend). This is another reason we don't spend too much time worrying about data compression and I/O bound workloads - when you've essentially expanded your available space by a factor of 4, you get the same benefits of compression, without wasting any…

If I can pluck your brain for a little, do you think LMDB would be a good option as a back end for time series analysis?

Re: RocksDB – A persistent key-value store for fast storage environments

#50

Post by rsynnott »

snewman wrote: post #22

Very nice work, and the wiki is also quite nice -- I wish more projects had a page like https://github.com/facebook/rocksdb/wiki/Rocksdb-Architectur... . It's really nice to see a clear, terse summary of what makes this project interesting relative to its predecessors. At my company (scalyr.com), we've built a more-or-less clone of LevelDB in Java, with a similar goal of extracting more performance on high-powered se…

> we've built a more-or-less clone of LevelDB in Java, with a similar goal of extracting more performance on high-powered servers (and better integration with our Java codebase).

This sounds quite interesting; have you considered open-sourcing it?

Post reply on HN