Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RocksDB – A persistent key-value store for fast storage environments

rocksdb.org

Post reply on HN
Print view View original

61–70 of 75 posts

Re: RocksDB – A persistent key-value store for fast storage environments

#61

Post by hyc_symas »

rossjudson wrote: post #53
hyc_symas wrote: post #40

Earlier quoted context omitted.

In our own workloads, writers are always going after the same pages in their index updates, which inevitably led to deadlocks in BerkeleyDB. As a result, we get much higher throughput with fully serialized writers than with "concurrent" writers. A microbench might show greater concurrency on simple write tasks, but in a real live system with elaborate schema, there's no payoff for us. As always, you have to profile y…

Is this the reason for your observation that LMDB is oriented towards read workloads? I can see how the extra code locking/concurrency code would expand the library size out of the CPU cache, though.

Yes, since readers don't require any locks at all and don't issue any blocking calls of any kind - syscalls, malloc, whatever - they run completely unimpeded. The moment you introduce fine-grained locks of any kind the overall performance (reads and writes) will decrease by at least an order of magnitude because readers will have to deal with lock contention.

Re: RocksDB – A persistent key-value store for fast storage environments

#62

Post by AaronFriel »

hyc_symas wrote: post #23
justin66 wrote: post #14

Earlier quoted context omitted.

Paging hyc_symas. Howard Chu isn't shy about talking about benchmark results and he can be found here and on twitter.

Hello hello! Pretty sure what actually happened in these is that the HDD's internal cache was still active, while the Crucial M4 SSD has no internal cache. The only other explanation is that I screwed up my partition offset on the SSD but I already double-checked that and the partitions were all 2MB aligned.

If HDD's internal cache was active how are synchronous writes still faster? Shouldn't the flush/sync ensure persistence of data?

If the cache was being used, the HDD results are not actually synchronous, a power loss event would result in data loss.

Re: RocksDB – A persistent key-value store for fast storage environments

#64

Post by techtalsky »

dhruba_b wrote: post #39
techtalsky wrote: post #16

Earlier quoted context omitted.

Can this be used as a drop-in replacement for LevelDB on queue technologies like ApolloMQ that use LevelDB as the default?

This can be used as a drop-in replacement. It disk format is compatible with leveldb. But once you upgrade to RocksDB, you won't be able to switch back to Leveldb (unless you restore ur data from a backup). One definite use-case for RocksDB is a message queue that has a high number of puts/gets/deletes. Also, please look at options.h to determine what options to tune. A not-tuned system will not show you much perf di…

Thank you. My company uses ApolloMQ but in a relatively low-data setting. (10-15 queues, with less than 5000 messages in a queue at a time) I don't know that it would make it worth it to move to a different data store but the idea was fascinating to me so I was curious. Thanks for the response.

Re: RocksDB – A persistent key-value store for fast storage environments

#65

Post by ashwinmurthy »

dhruba_b wrote: post #41
jonstewart wrote: post #18

Earlier quoted context omitted.

The primary enhancements over LevelDB seem to be parallel compactions of disjoint ranges, to take advantage of cheap seeks on flash storage, and the ability to parameterize core algorithms and data structures to suit a particular anticipated workload. All very cool; anything else major? Also, there aren't JNI bindings... are there? Thanks for the contribution. Just started using LevelDB on a project, but deployment w…

Thanks for your comments Jon. RocksDB shares some of its genes with LevelDB.. something like a parent-child relationship. Please check out Universal Comaction Style, multi-threaded-compaction, pipelined memtables. I used to have JNI bindings that I pulled in from https://github.com/fusesource/leveldbjni but it was difficult for me to update the JNI everytime we added new apis to RocksDB. It would be great if somebody…

Can it be configured as a distributed No SQL database like Cassandra?

Re: RocksDB – A persistent key-value store for fast storage environments

#66

Post by ExpiredLink »

Patient0 wrote: post #48

I'm surprised that the C++ code is not using the RAII idiom in some obvious places. For example: https://github.com/facebook/rocksdb/blob/master/db/db_impl.c There are many places with bracketed calls to mutex_.Lock and mutex_.Unlock(). An example: mutex_.Unlock(); LogFlush(options_.info_log); env_->SleepForMicroseconds(1000000); mutex_.Lock() Why didn't the authors use the RAII idiom here? Even if there are no excep…

A lot of C++ code essentially is 'C with classes': no RAII, no exception handling.

Re: RocksDB – A persistent key-value store for fast storage environments

#67

Post by kodablah »

snewman wrote: post #54
rsynnott wrote: post #50

Earlier quoted context omitted.

> we've built a more-or-less clone of LevelDB in Java, with a similar goal of extracting more performance on high-powered servers (and better integration with our Java codebase). This sounds quite interesting; have you considered open-sourcing it?

Yes, we'd like to release it someday, but it won't be any time soon unfortunately. There are a lot of dependencies on other parts of our codebase, e.g. for configuration and monitoring, which would need to be cleaned up. We will probably at least publish a report describing the work in more detail, some time in the next few months, on our blog ( http://blog.scalyr.com ).

It would be helpful if this description also includes a comparison with http://www.mapdb.org/, I am curious the differences (beyond just the ecosystem around the DB)

Re: RocksDB – A persistent key-value store for fast storage environments

#68

Post by phunge »

tsewlliw wrote: post #55
Patient0 wrote: post #48

I'm surprised that the C++ code is not using the RAII idiom in some obvious places. For example: https://github.com/facebook/rocksdb/blob/master/db/db_impl.c There are many places with bracketed calls to mutex_.Lock and mutex_.Unlock(). An example: mutex_.Unlock(); LogFlush(options_.info_log); env_->SleepForMicroseconds(1000000); mutex_.Lock() Why didn't the authors use the RAII idiom here? Even if there are no excep…

fixed your link: https://github.com/facebook/rocksdb/blob/master/db/db_impl.c... Take another look! There's a guard object used at the function scope to ensure the lock is released, and this block is bracketed to release and reacquire the lock, not acquire and release. There may be a case for a guard object that does the release/reacquire, but its definitely not a slam dunk like acquire/release

Still, that's not exception-safe, correct? If LogFlush or SleepForMicroseconds throws an exception the mutex will be unlocked twice, which pthreads disallows for normal mutexes...

Re: RocksDB – A persistent key-value store for fast storage environments

#69

Post by bjconlan »

hyc_symas wrote: post #26
snewman wrote: post #22

Very nice work, and the wiki is also quite nice -- I wish more projects had a page like https://github.com/facebook/rocksdb/wiki/Rocksdb-Architectur... . It's really nice to see a clear, terse summary of what makes this project interesting relative to its predecessors. At my company (scalyr.com), we've built a more-or-less clone of LevelDB in Java, with a similar goal of extracting more performance on high-powered se…

SkyDB using LMDB gets 3GB/sec on a standalone PC. https://groups.google.com/forum/#!msg/skydb/CMKQSLf2WAw/zBO1...

Wow, Awesome link, LMDB always seems to fly under the radar, SkyDB+LMDB. Genius. (and written in go! I'm sold... well will at least give it a bash)

Re: RocksDB – A persistent key-value store for fast storage environments

#70

Post by waffleclub »

jbapple wrote: post #44
dhruba_b wrote: post #35

Earlier quoted context omitted.

RocksDB has an LSM architecture, similar in nature to HBase, leveldb, etc. But the implementation is based on a Theorem that we will be publishing shortly. I am working on the Theorem with a colleague of mine. Cache Oblivious B-trees is an interesting paper. Similarly fractal trees. Most of them optimize the case when index nodes are not in memory. However, in our use-cases, we typically configure the system in such…

Can you share with us the statement of that theorem? What is "UniversalStyleCompaction", and why is it capitalized and missing spaces? How does a Bloom filter for range scans work? Standard Bloom filters (as you know) are for existence only.

I'm guessing this may have been an early draft of some of the statements of the theorem:

http://webcache.googleusercontent.com/search?q=cache:fTxlRmb...

Post reply on HN