Earlier quoted context omitted.
It is limited to the logical address space. Since most current x86-64 machines have only 48bit address space, 256TB, and assuming the kernel keeps half of the space for itself, then yes, the current limit is 128TB. But I suspect we'll be seeing 56bit address spaces fairly soon. It is a single-writer DB, one DB-wide writer lock. Fine-grained locking is a tar pit.
Makes sense. Most impressive about LMDB to me is the zero-copy model for readers, with is no extra memcpy needed, maybe that is something obvious for database gurus but it is pretty clever trick I think.
RocksDB – A persistent key-value store for fast storage environments
31–40 of 75 posts
Re: RocksDB – A persistent key-value store for fast storage environments
#32Earlier quoted context omitted.
The LMDB statistics are very strange - why is synchronous SSD performance worse on most figures than HDD performance? Something seems very wrong with these benchmarks: Section 5 (SSD) F (Synchronous Writes) Random Writes LevelDB 342 ops/sec Kyoto TreeDB 67 ops/sec SQLite3 114 ops/sec MDB 148 ops/sec MDB, no MetaSync 322 ops/sec BerkeleyDB 291 ops/sec Section 8 (HDD) F (Synchronous Writes) Random Writes LevelDB 1291 o…
Keep in mind, the HDD was using ext2 and the SSD was using reiserfs. Synchronous writes on ext2 are faster than all journaling filesystems.
Re: RocksDB – A persistent key-value store for fast storage environments
#33Earlier quoted context omitted.
The LMDB statistics are very strange - why is synchronous SSD performance worse on most figures than HDD performance? Something seems very wrong with these benchmarks: Section 5 (SSD) F (Synchronous Writes) Random Writes LevelDB 342 ops/sec Kyoto TreeDB 67 ops/sec SQLite3 114 ops/sec MDB 148 ops/sec MDB, no MetaSync 322 ops/sec BerkeleyDB 291 ops/sec Section 8 (HDD) F (Synchronous Writes) Random Writes LevelDB 1291 o…
> The LMDB statistics are very strange - why is synchronous SSD performance worse on most figures than HDD performance? Could it be that most database engines are based on algorithms that were developed before SSDs were significant, and were extremely optimized for HDD performance?
Re: RocksDB – A persistent key-value store for fast storage environments
#34Earlier quoted context omitted.
Howard, is LMDB effectively limited to 128T (on 64bit machines and 2GB on 32bit ones, not that one should be running large databases on 32bit machines)? Also what about concurrent writes? Does it have a database wide writer lock or is it per key (per page?) ?
It is limited to the logical address space. Since most current x86-64 machines have only 48bit address space, 256TB, and assuming the kernel keeps half of the space for itself, then yes, the current limit is 128TB. But I suspect we'll be seeing 56bit address spaces fairly soon. It is a single-writer DB, one DB-wide writer lock. Fine-grained locking is a tar pit.
We (Tokutek) tried for a long time to get by with a big monolithic lock, and a) competing with InnoDB was really hard since they do concurrent writers really really well, and b) when we did decide to break up the lock, it wasn't as hard as we thought it would be and it worked really really well.
Don't get discouraged, break up that lock!
Re: RocksDB – A persistent key-value store for fast storage environments
#35Hi guys, I am Dhruba and I work in the Database Engineering team at Facebook. We just released RocksDB as an open source project. If anybody has any technical questions about RocksDB, please feel free to ask. Thanks.
Hi Dhruba, thanks for volunteering to ask questions. What are the big algorithmic ideas behind RocksDB? My understanding is that LevelDB is based on log structured merge trees. These can be deamortized using methods from Overmars's "The Design of Dynamic Data Structures" or Bender et al.'s "Cache-Oblivious Streaming B-trees". How did you reduce latency? What else was slowing down databases larger than RAM? How did yo…
Cache Oblivious B-trees is an interesting paper. Similarly fractal trees. Most of them optimize the case when index nodes are not in memory. However, in our use-cases, we typically configure the system in such a way that most index nodes are in memory.
For an LSM database, the key component is "compaction". You can ingest data only as fast as you can compact, otherwise u get a unstable system. .1 RocksDB replaced the Level-style compaction of leveldb with UniversalStyleCompaction that has reduced write amplification. This boosts performance. 2. RocksDB implemented multi-threaded write, which means that parallel compactions on different parts of the database can occur simultaneously. This boosts performance. 3. Bloom filter for range-scans: this boost read performance 4. MergeType records that allows higher level objects (counters, lists) use only-write instead of a read-modify-write. Improves performance. 5. And many more...
Re: RocksDB – A persistent key-value store for fast storage environments
#36this is cool, though I'd wonder how it compares to Kyoto Cabinet. another big issue I've run into personally is the fact that both LevelDB and KC don't explicitly support multiple processes reading the db at once. (KC's API allows this but advises against it, LevelDB afaik doesn't even allow it.) I wonder if RocksDB gets past this.
Kyoto Cabinet will self-corrupt if you use it that way. LMDB supports multi-process explicitly.
Re: RocksDB – A persistent key-value store for fast storage environments
#37Earlier quoted context omitted.
Keep in mind, the HDD was using ext2 and the SSD was using reiserfs. Synchronous writes on ext2 are faster than all journaling filesystems.
Not three orders of magnitude faster, which is the difference between hdd and ssd random writes.
Re: RocksDB – A persistent key-value store for fast storage environments
#38Hi guys, I am Dhruba and I work in the Database Engineering team at Facebook. We just released RocksDB as an open source project. If anybody has any technical questions about RocksDB, please feel free to ask. Thanks.
How much do you think RocksDB/LevelDB performance is impacted by the use of relatively coarse-grained locking? Another LevelDB fork, HyperLevelDB [1] implemented a fine grained scheme with performance benefits. Disclaimer: I am working on a (unreleased) fork of LevelDB that uses hardware transactional memory for synchronization using the new TSX-NI instructions present on Haswell processors. [1] - http://hyperdex.org…
Re: RocksDB – A persistent key-value store for fast storage environments
#39Hi guys, I am Dhruba and I work in the Database Engineering team at Facebook. We just released RocksDB as an open source project. If anybody has any technical questions about RocksDB, please feel free to ask. Thanks.
Can this be used as a drop-in replacement for LevelDB on queue technologies like ApolloMQ that use LevelDB as the default?
One definite use-case for RocksDB is a message queue that has a high number of puts/gets/deletes.
Also, please look at options.h to determine what options to tune. A not-tuned system will not show you much perf difference from leveldb.
Re: RocksDB – A persistent key-value store for fast storage environments
#40Earlier quoted context omitted.
It is limited to the logical address space. Since most current x86-64 machines have only 48bit address space, 256TB, and assuming the kernel keeps half of the space for itself, then yes, the current limit is 128TB. But I suspect we'll be seeing 56bit address spaces fairly soon. It is a single-writer DB, one DB-wide writer lock. Fine-grained locking is a tar pit.
Fine-grained locking is hard , but "tar pit" is unfair and honestly a bad attitude. It's crucial for modern applications, and it can be done if you're careful, and it can be done really well. We (Tokutek) tried for a long time to get by with a big monolithic lock, and a) competing with InnoDB was really hard since they do concurrent writers really really well, and b) when we did decide to break up the lock, it wasn't…
As always, you have to profile your workload and see where the delays and bottlenecks really are. Taking a single mutex instead of continuously locking/unlocking all over the place was a win for us.