[deleted]
Design Review: Key-Value Storage
41–50 of 90 posts
Re: Design Review: Key-Value Storage
#42The page writes that LMDB "is exceptionally fast for our kinds of load", and then links to an in-memory microbenchmark: http://www.lmdb.tech/bench/inmem/ Aren't they interested in persistence of the key-value data? In my experience, once data is persisted to disk or SSD, LMDB is way slower from alternatives because it needs to operate in synchronous mode to avoid corruption (effectively flushing to disk after after e…
We have only ever compared LMDB in synchronous mode to other DBs in synchronous mode, and LMDB in asynch mode to other DBs in asynch mode. Come on, that's too obvious.
And LMDB beats the crap out of SQLite, in any mode. http://www.lmdb.tech/bench/microbench/
Replacing SQLite's Btree engine with LMDB makes the SQLite footprint smaller, faster, and more reliable too. https://github.com/LMDB/sqlightning
Re: Design Review: Key-Value Storage
#43The page writes that LMDB "is exceptionally fast for our kinds of load", and then links to an in-memory microbenchmark: http://www.lmdb.tech/bench/inmem/ Aren't they interested in persistence of the key-value data? In my experience, once data is persisted to disk or SSD, LMDB is way slower from alternatives because it needs to operate in synchronous mode to avoid corruption (effectively flushing to disk after after e…
We have only ever compared LMDB in synchronous mode to other DBs in synchronous mode, and LMDB in asynch mode to other DBs in asynch mode. Come on, that's too obvious. And LMDB beats the crap out of SQLite, in any mode. http://www.lmdb.tech/bench/microbench/ Replacing SQLite's Btree engine with LMDB makes the SQLite footprint smaller, faster, and more reliable too. https://github.com/LMDB/sqlightning
It's not that simple. What you mean by saying "(a)synchronous mode" is very different from database to database. See my above comment. Is SQLite synchronous or asynchronous if you configure it as journal_mode=WAL and PRAGMA synchronous=NORMAL?
For my purposes as an application developer, I care about comparing databases operating in safe mode i.e. a system crash should never cause total data loss. According to my experience SQLite's safe mode is many times faster than LMDB's safe mode, with a write workload. While LMDB is thrashing the disk and achieves only a handful of write transactions per second.
Re: Design Review: Key-Value Storage
#44The page writes that LMDB "is exceptionally fast for our kinds of load", and then links to an in-memory microbenchmark: http://www.lmdb.tech/bench/inmem/ Aren't they interested in persistence of the key-value data? In my experience, once data is persisted to disk or SSD, LMDB is way slower from alternatives because it needs to operate in synchronous mode to avoid corruption (effectively flushing to disk after after e…
I have been comparing multiple key-value stores and lmdb has been the simplest and fastest to use with or without MDB_NOSYNC. It beats almost every kv store in writes.. and in reads it is untouchable by anything by a large magnitude.
Re: Design Review: Key-Value Storage
#45The page writes that LMDB "is exceptionally fast for our kinds of load", and then links to an in-memory microbenchmark: http://www.lmdb.tech/bench/inmem/ Aren't they interested in persistence of the key-value data? In my experience, once data is persisted to disk or SSD, LMDB is way slower from alternatives because it needs to operate in synchronous mode to avoid corruption (effectively flushing to disk after after e…
Where on this page does it say that LMDB was run with MDB_NOSYNC? Tests were run with the DB on a tempfs and then again (lower down on the page) on ext4.
Re: Design Review: Key-Value Storage
#46The page writes that LMDB "is exceptionally fast for our kinds of load", and then links to an in-memory microbenchmark: http://www.lmdb.tech/bench/inmem/ Aren't they interested in persistence of the key-value data? In my experience, once data is persisted to disk or SSD, LMDB is way slower from alternatives because it needs to operate in synchronous mode to avoid corruption (effectively flushing to disk after after e…
We have only ever compared LMDB in synchronous mode to other DBs in synchronous mode, and LMDB in asynch mode to other DBs in asynch mode. Come on, that's too obvious. And LMDB beats the crap out of SQLite, in any mode. http://www.lmdb.tech/bench/microbench/ Replacing SQLite's Btree engine with LMDB makes the SQLite footprint smaller, faster, and more reliable too. https://github.com/LMDB/sqlightning
> And LMDB beats the crap out of SQLite, in any mode. http://www.lmdb.tech/bench/microbench/
Alright, I went through this page to see why my experience is different. While it mentions that they enabled SQLite's WAL journal, it also mentions that the synchronous writes were performed with PRAGMA synchronous=FULL.
I believe that if they set PRAGMA synchronous=NORMAL, they will get an ACI-reliable database that is way faster than LMDB with write-to-disk workloads.
Most of the measurements on that page are not really useful to me; they do not persist the data (db on tmpfs) or they do not care about reliability, using dangerous settings. Only few measurements are both persisting the data and writing safely, but the SQLite configuration is sub-optimal for them.
Re: Design Review: Key-Value Storage
#47The page writes that LMDB "is exceptionally fast for our kinds of load", and then links to an in-memory microbenchmark: http://www.lmdb.tech/bench/inmem/ Aren't they interested in persistence of the key-value data? In my experience, once data is persisted to disk or SSD, LMDB is way slower from alternatives because it needs to operate in synchronous mode to avoid corruption (effectively flushing to disk after after e…
Where on this page does it say that LMDB was run with MDB_NOSYNC? Tests were run with the DB on a tempfs and then again (lower down on the page) on ext4.
Section 4:
"using a 512GB Samsung 830 SSD and an ext4 partition.
The actual drive characteristics should not matter because the test datasets still fit entirely in RAM and are all using asynchronous writes"
Re: Design Review: Key-Value Storage
#48Curious about how this compares to TiKV...
Re: Design Review: Key-Value Storage
#49The page writes that LMDB "is exceptionally fast for our kinds of load", and then links to an in-memory microbenchmark: http://www.lmdb.tech/bench/inmem/ Aren't they interested in persistence of the key-value data? In my experience, once data is persisted to disk or SSD, LMDB is way slower from alternatives because it needs to operate in synchronous mode to avoid corruption (effectively flushing to disk after after e…
I have been comparing multiple key-value stores and lmdb has been the simplest and fastest to use with or without MDB_NOSYNC. It beats almost every kv store in writes.. and in reads it is untouchable by anything by a large magnitude.
Agreed for the read workloads, I have the same experience. It was designed primarily for an LDAP directory after all, an application that is very read-heavy.
Re: Design Review: Key-Value Storage
#50I have a layman question if somebody could please answer. I have never in my entire life seen databases fail. But db failures and issues seem to be brought up all the time. Now I understand that part if this maybe the cost function associated with them. But I'm sure there's also something that I have no clue about. So my questions are:
1) what kind of problems do databases actually face.
2) what kind of scenarios create those problems.
3) how does a programmer go about testing them?