Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RocksDB – A persistent key-value store for fast storage environments

rocksdb.org

Post reply on HN
Print view View original

51–60 of 75 posts

Re: RocksDB – A persistent key-value store for fast storage environments

#51

Post by rossjudson »

hyc_symas wrote: post #23
justin66 wrote: post #14

Earlier quoted context omitted.

Paging hyc_symas. Howard Chu isn't shy about talking about benchmark results and he can be found here and on twitter.

Hello hello! Pretty sure what actually happened in these is that the HDD's internal cache was still active, while the Crucial M4 SSD has no internal cache. The only other explanation is that I screwed up my partition offset on the SSD but I already double-checked that and the partitions were all 2MB aligned.

I'm going to compile up LMDB and bench it on a 96GB DL380g8 with quad 3TB ioDrive-2s. Should be interesting to see how various database sizes play out, and what the write amp looks like. I am not seeing much about LMDB's NUMA awareness -- guess I need to keep digging.

Re: RocksDB – A persistent key-value store for fast storage environments

#52

Post by jbellis »

ithkuil wrote: post #43
jbellis wrote: post #32

Earlier quoted context omitted.

Not three orders of magnitude faster, which is the difference between hdd and ssd random writes.

Three orders of magnitude faster would mean 1000x faster. You probably meant 3 times faster.

SSDs really are 1000x faster at random writes (~200,000 iops vs ~200 iops)

Re: RocksDB – A persistent key-value store for fast storage environments

#53

Post by rossjudson »

hyc_symas wrote: post #40
leif wrote: post #34

Earlier quoted context omitted.

Fine-grained locking is hard , but "tar pit" is unfair and honestly a bad attitude. It's crucial for modern applications, and it can be done if you're careful, and it can be done really well. We (Tokutek) tried for a long time to get by with a big monolithic lock, and a) competing with InnoDB was really hard since they do concurrent writers really really well, and b) when we did decide to break up the lock, it wasn't…

In our own workloads, writers are always going after the same pages in their index updates, which inevitably led to deadlocks in BerkeleyDB. As a result, we get much higher throughput with fully serialized writers than with "concurrent" writers. A microbench might show greater concurrency on simple write tasks, but in a real live system with elaborate schema, there's no payoff for us. As always, you have to profile y…

Is this the reason for your observation that LMDB is oriented towards read workloads?

I can see how the extra code locking/concurrency code would expand the library size out of the CPU cache, though.

Re: RocksDB – A persistent key-value store for fast storage environments

#54

Post by snewman »

rsynnott wrote: post #50
snewman wrote: post #22

Very nice work, and the wiki is also quite nice -- I wish more projects had a page like https://github.com/facebook/rocksdb/wiki/Rocksdb-Architectur... . It's really nice to see a clear, terse summary of what makes this project interesting relative to its predecessors. At my company (scalyr.com), we've built a more-or-less clone of LevelDB in Java, with a similar goal of extracting more performance on high-powered se…

> we've built a more-or-less clone of LevelDB in Java, with a similar goal of extracting more performance on high-powered servers (and better integration with our Java codebase). This sounds quite interesting; have you considered open-sourcing it?

Yes, we'd like to release it someday, but it won't be any time soon unfortunately. There are a lot of dependencies on other parts of our codebase, e.g. for configuration and monitoring, which would need to be cleaned up.

We will probably at least publish a report describing the work in more detail, some time in the next few months, on our blog (http://blog.scalyr.com).

Re: RocksDB – A persistent key-value store for fast storage environments

#55

Post by tsewlliw »

Patient0 wrote: post #48

I'm surprised that the C++ code is not using the RAII idiom in some obvious places. For example: https://github.com/facebook/rocksdb/blob/master/db/db_impl.c There are many places with bracketed calls to mutex_.Lock and mutex_.Unlock(). An example: mutex_.Unlock(); LogFlush(options_.info_log); env_->SleepForMicroseconds(1000000); mutex_.Lock() Why didn't the authors use the RAII idiom here? Even if there are no excep…

fixed your link: https://github.com/facebook/rocksdb/blob/master/db/db_impl.c...

Take another look! There's a guard object used at the function scope to ensure the lock is released, and this block is bracketed to release and reacquire the lock, not acquire and release. There may be a case for a guard object that does the release/reacquire, but its definitely not a slam dunk like acquire/release

Re: RocksDB – A persistent key-value store for fast storage environments

#56

Post by e12e »

Patient0 wrote: post #48

I'm surprised that the C++ code is not using the RAII idiom in some obvious places. For example: https://github.com/facebook/rocksdb/blob/master/db/db_impl.c There are many places with bracketed calls to mutex_.Lock and mutex_.Unlock(). An example: mutex_.Unlock(); LogFlush(options_.info_log); env_->SleepForMicroseconds(1000000); mutex_.Lock() Why didn't the authors use the RAII idiom here? Even if there are no excep…

That link should be:

https://github.com/facebook/rocksdb/blob/master/db/db_impl.c...

(not db_impl.c but .cc) I was wondering for a while if it was possible to do RAII in idiomatic c99 -- and it appears it isn't (or doesn't make as much sense, anyway).

Re: RocksDB – A persistent key-value store for fast storage environments

#57

Post by hyc_symas »

vicaya wrote: post #46
hyc_symas wrote: post #27

Earlier quoted context omitted.

LSMs have a long long way to go to catch up to LMDB. http://symas.com/mdb/hyperdex/

For reads, sure. LSMs are optimized for writes, while LMDB, which is a nice B-tree implementation is optimized for reads. LSMs are getting popular because it's harder to scale durable writes than reads, which can be handled (in many cases independently) by caching.

LSMs are solving a problem that is rapidly becoming irrelevant due to the multiple NVRAM technologies entering the market. With NVDIMMs, MRAM, FeRAM, PRAM, etc., all your writes can be durable for free. And if you'll notice in that HyperDex benchmark I posted, the LSM write performance was still worse than LMDB, while wasting several times more CPU.

Going forward, power efficiency will still be crucial - for as long as civilization persists. But optimizing durable writes will be about as useful as optimizing delay loops.

Re: RocksDB – A persistent key-value store for fast storage environments

#58

Post by wbolster »

The benchmark at https://github.com/facebook/rocksdb/wiki/Performance-Benchma... states that for LevelDB, "in 24 hours it inserted only 2 million key-values", and that "each key is of size 10 bytes, each value is of size 800 bytes".

I might be missing something, but that took just a few minutes on my ~2 year old desktop machine. Sample code: https://gist.github.com/wbolster/7487225

Re: RocksDB – A persistent key-value store for fast storage environments

#59

Post by hyc_symas »

kitsune_ wrote: post #49
hyc_symas wrote: post #31

Earlier quoted context omitted.

It's pretty significant, yes. Eliminating multiple copies of everything got us a 4:1 reduction in memory footprint in OpenLDAP slapd (compared to our BerkeleyDB-based backend). This is another reason we don't spend too much time worrying about data compression and I/O bound workloads - when you've essentially expanded your available space by a factor of 4, you get the same benefits of compression, without wasting any…

If I can pluck your brain for a little, do you think LMDB would be a good option as a back end for time series analysis?

I'm sorry, I'm not familiar enough with the workload to answer that. If you're primarily doing sequential writes, it seems like it could work well for it.

Re: RocksDB – A persistent key-value store for fast storage environments

#60

Post by hyc_symas »

rossjudson wrote: post #51
hyc_symas wrote: post #23

Earlier quoted context omitted.

Hello hello! Pretty sure what actually happened in these is that the HDD's internal cache was still active, while the Crucial M4 SSD has no internal cache. The only other explanation is that I screwed up my partition offset on the SSD but I already double-checked that and the partitions were all 2MB aligned.

I'm going to compile up LMDB and bench it on a 96GB DL380g8 with quad 3TB ioDrive-2s. Should be interesting to see how various database sizes play out, and what the write amp looks like. I am not seeing much about LMDB's NUMA awareness -- guess I need to keep digging.

For reads we get linear scaling out to 64 cores. Using cache-aligned data structures plays a big part in that for NUMA. (At the moment that's the largest machine we have in our lab.) For writes, there's basically no scaling. Write amplification is logN, proportional to tree height.
Post reply on HN