Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Design Review: Key-Value Storage

mozilla.github.io

Post reply on HN
Print view View original

81–90 of 90 posts

Re: Design Review: Key-Value Storage

#81

Post by mamcx »

hyc_symas wrote: post #40
mamcx wrote: post #9

I see their have some questions (like how good is the windows and android support) that are not answered? or only internally? I think will be good to see what them found.

Windows and Android are fully supported, have been for years. As are iOS and MacOSX and all the BSDs.

Thats great to know, and I suspect that. But what I'm saying is that them ask some valid questions but where are the results of their findings?

Because if I read this, I could conclude LMDB could have troubles in that areas...

Re: Design Review: Key-Value Storage

#82

Post by hyc_symas »

jimis wrote: post #55
hyc_symas wrote: post #51

Earlier quoted context omitted.

SQLite's safe mode is not comparable to LMDB's; SQLite is vulnerable to silent data loss in a crash. https://wisdom.cs.wisc.edu/workshops/spring-14/talks/Thanu.p... LMDB is not.

Isn't this from the paper "all filesystems are not created equal"? [1] If you search the tables for "sqlite-wal" you will see that it shows zero vulnerabilities. [1] https://www.usenix.org/system/files/conference/osdi14/osdi14...

Ah, the earlier result was for SQLite-rollback, their default mode. IMO any DB should default to its safest mode.

Re: Design Review: Key-Value Storage

#83

Post by hyc_symas »

justin66 wrote: post #78
hyc_symas wrote: post #67

Earlier quoted context omitted.

The main reason for that warning about NFS is that people will try it, see that it seems to work, and then get careless and try to use the same DB from two different hosts at once. It's inevitable when you're working on files living on networked filesystems, and it cannot work. NFS doesn't offer any cache coherency guarantees, and the mutexes used for synchronizing writers only work on the host that created them. Not…

If we're using lmdb for something that's largely or entirely read only, how viable is using it on NFS with (shudder) lock files or something like that?

If it's 100% read-only, you could probably use it safely. Make sure its filesystem on the NFS server is mounted read-only, and obviously all the clients' NFS mounts must also be read-only. As for "largely read only" - if there are any writers at all, all bets are off.

LMDB automatically detects read-only filesystems, and turns off its locking in that case, so it should perform as well as anyone could expect NFS to perform.

Re: Design Review: Key-Value Storage

#84

Post by hyc_symas »

jnwatson wrote: post #69
acqq wrote: post #63

Earlier quoted context omitted.

So the endianness answer is, from your link: "They are still byte-order dependent but almost everyone uses little-endian CPUs these days so it's not much of an issue." Which just means "we solve the problem by ignoring it completely." And what about the limitations on the 32-bit system? Isn't there also needed to use memory space of RAM for the complete size of the database, that's how that database works if I unders…

The last issue is merely one of performance, not capability. I’ve successfully used LMDB in 32-bit land, though it takes some effort. I had to scan available virtual memory for the largest contiguous chunk and use that. Bigger issues are growing the database size and performance in low memory. No transactions can be running to increase the map size, which is usually hard to coordinate. Also, in low memory situations,…

Fwiw, we've benchmarked on Raspberry Pis with slow SDcards. You want to talk about low RAM situations and slow I/Os. The reality is still the same though - every other DB engine is many times slower under the same conditions.

Even when the DB is 5x or 50x larger than RAM... http://www.lmdb.tech/bench/hyperdex/

Re: Design Review: Key-Value Storage

#85

Post by guachesuedehack »

mrjn wrote: post #13
coleifer wrote: post #7

Earlier quoted context omitted.

Also typically more complicated and require a separate compaction process. They're good for writing lots of data, but not so great for random reads.

Author of Badger here. Our design of separating keys and values has gotten us incredibly fast writes, while still keeping the read latencies neck-to-neck against B+ trees. Worth checking out: https://github.com/dgraph-io/badger

Nice, why the choice of go ?

Re: Design Review: Key-Value Storage

#86

Post by catwell »

hyc_symas wrote: post #67
catwell wrote: post #65

In case someone involved in this reads this thread: the document does not specify which LMDB version you tested. I suggest you run your tests with the `mdb.master` branch, i.e. the work towards a future 1.0, and not the stable 0.9 branch. The answers to several of your interrogations will depend on that: with `mdb.master` you can use the VL32 mode which greatly improves usage on 32 bit platforms, and Windows support…

The main reason for that warning about NFS is that people will try it, see that it seems to work, and then get careless and try to use the same DB from two different hosts at once. It's inevitable when you're working on files living on networked filesystems, and it cannot work. NFS doesn't offer any cache coherency guarantees, and the mutexes used for synchronizing writers only work on the host that created them. Not…

Things like closing DBIs, transactions and cursors in the wrong order, for instance.

Re: Design Review: Key-Value Storage

#87

Post by adobeeee »

tfha wrote: post #80
adobeeee wrote: post #50

I have a layman question if somebody could please answer. I have never in my entire life seen databases fail. But db failures and issues seem to be brought up all the time. Now I understand that part if this maybe the cost function associated with them. But I'm sure there's also something that I have no clue about. So my questions are: 1) what kind of problems do databases actually face. 2) what kind of scenarios cre…

We ship a consumer application with databases in it (multiple). Even the ACID ones fail all the time, maybe 1-3% of our userbase has had a corruption at some point.

Can you talk about the reasons of the failure?

Re: Design Review: Key-Value Storage

#88

Post by hyc_symas »

catwell wrote: post #86
hyc_symas wrote: post #67

Earlier quoted context omitted.

The main reason for that warning about NFS is that people will try it, see that it seems to work, and then get careless and try to use the same DB from two different hosts at once. It's inevitable when you're working on files living on networked filesystems, and it cannot work. NFS doesn't offer any cache coherency guarantees, and the mutexes used for synchronizing writers only work on the host that created them. Not…

Things like closing DBIs, transactions and cursors in the wrong order, for instance.

That won't corrupt the DB. It might corrupt the heap of your current process, but the DB won't be harmed.

C doesn't have destructors. If you free stuff in the wrong order, you lose. That's nothing to do with LMDB's API. Even if LMDB cleaned everything up internally, you'd still have dangling pointers in your app space. No language or API can prevent that.

The only thing you can do in the API that might possibly corrupt the DB is to use MDB_RESERVE and then store a value bigger than the space you reserved. That generally causes a SEGV, and you'll discover very quickly that you have a bug in your code. LMDB will fail fast, every time, and every failure will be a bug in your own code. Makes debugging very quick.

Re: Design Review: Key-Value Storage

#89

Post by floatingatoll »

romed wrote: post #57
floatingatoll wrote: post #36

Earlier quoted context omitted.

Transactions let you perform any series of SQL commands with various expectations around data safety and locking guarantees depending on isolation level. Batch writes provide a tiny subset of the full possibilities of transactions. While sufficient in many cases, that cannot be generalized to "LevelDB supports transactions".

That's just your pet definition of transaction. That is not universally accepted.

Universal acceptance is not a valid criterion for human beings at scale.

Schizophrenic people don’t accept facts like “this wall exists”, yet we still reach agreement as a society that a wall exists and don’t try to walk through walls because they exist.

You are of course welcome to hold any belief you wish, but believing something that puts you in direct contradiction to an entire industry significantly raises the bar of proof you must provide in order to convince others to listen to you.

Your reply does not provide that proof, and thus your argument is not persuasive.

Re: Design Review: Key-Value Storage

#90

Post by catwell »

hyc_symas wrote: post #88
catwell wrote: post #86

Earlier quoted context omitted.

Things like closing DBIs, transactions and cursors in the wrong order, for instance.

That won't corrupt the DB. It might corrupt the heap of your current process, but the DB won't be harmed. C doesn't have destructors. If you free stuff in the wrong order, you lose. That's nothing to do with LMDB's API. Even if LMDB cleaned everything up internally, you'd still have dangling pointers in your app space. No language or API can prevent that. The only thing you can do in the API that might possibly corru…

Glad to hear that.

To be honest the only real corruption issues I had with LMDB in practice were dues to me using MDB_NOSYNC, and broken mmap behavior on some Android devices (on external storage).

Post reply on HN