Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Design Review: Key-Value Storage

mozilla.github.io

Post reply on HN
Print view View original

61–70 of 90 posts

Re: Design Review: Key-Value Storage

#61

Post by Twirrim »

romed wrote: post #57
floatingatoll wrote: post #36

Earlier quoted context omitted.

Transactions let you perform any series of SQL commands with various expectations around data safety and locking guarantees depending on isolation level. Batch writes provide a tiny subset of the full possibilities of transactions. While sufficient in many cases, that cannot be generalized to "LevelDB supports transactions".

That's just your pet definition of transaction. That is not universally accepted.

I would love to see an example of someone actually knowledgeable about databases who has a different definition for transaction.

Re: Design Review: Key-Value Storage

#62

Post by philix001 »

adobeeee wrote: post #60
mbreese wrote: post #53

Earlier quoted context omitted.

The easiest scenario to imagine is a hardware failure or power outage. The database was in the middle of doing something, and then was prevented by a hard drive dying or the lights going out. One way to test such a thing is to literally unplug the computer to see how it handles the failure. So, let's say you have a client/server application... the client is telling the server (database) to write some records to the d…

Thank you for explaining so well and clearly! To you and others, are there any other scenarios too that happen in production?

Enough material here to scare anyone about databases

https://jepsen.io/talks

Re: Design Review: Key-Value Storage

#63

Post by acqq »

mykmelez wrote: post #23
acqq wrote: post #18

I thought that the issues like this would be blocking: https://stackoverflow.com/questions/44407659/how-to-force-64... Who knows how it should work on 32-bit systems? And isn't endianess also a problem? And doesn't SQLite solve both problems by default? And isn't also possible to configure SQLite to be very fast, if one know what he's doing? Btw I'd expected that the "notes here" https://docs.google.com/document/d/1b…

According to @hyc in https://monero.stackexchange.com/questions/2606/are-the-lmdb... , "As of v0.10.0, yes the LMDB files are cross-compatible between 32 and 64bit architectures. They have always been cross-compatible between OSs. They are still byte-order dependent but almost everyone uses little-endian CPUs these days so it's not much of an issue."

So the endianness answer is, from your link: "They are still byte-order dependent but almost everyone uses little-endian CPUs these days so it's not much of an issue." Which just means "we solve the problem by ignoring it completely."

And what about the limitations on the 32-bit system? Isn't there also needed to use memory space of RAM for the complete size of the database, that's how that database works if I understood? Which makes LMDB effectively unsuitable for 32-bit systems:

https://stackoverflow.com/questions/52862176/lmdb-open-large...

"your user will need to either enable PAE on their system, or upgrade to 64-bit CPU. If neither of these is an option in your application, then you cannot use a memory mapped file larger than your available address space"

In short it still looks that LMDB is designed effectively only for one-endianness and only for 64-bit systems, which is still limiting for many use cases.

Again, it seems that for LMDB is again "solving" the problem by ignoring it. Which is OK if it fits your use case... But shouldn't Firefox work properly on 32-bit systems? Or did they completely decide that they don't want to target any 32-bit platform anymore?

Moreover, there are use-case scenarios where the memory mapped file approach can suffer from problems of unnecessarily reading the page that will anyway be completely overwritten. My conclusion is... if LMDB "works for you" fine, but do properly your research first. I still believe SQLite covers much more use cases and is safer starting point for many of them, including Firefox use cases, until I read that they really decided to reduce these.

Re: Design Review: Key-Value Storage

#64

Post by erpellan »

Twirrim wrote: post #61
romed wrote: post #57

Earlier quoted context omitted.

That's just your pet definition of transaction. That is not universally accepted.

I would love to see an example of someone actually knowledgeable about databases who has a different definition for transaction.

Atomicity, consistency, isolation, durability.

https://en.wikipedia.org/wiki/ACID_(computer_science)

But that's just like, the industry's opinion, man.

Re: Design Review: Key-Value Storage

#65

Post by catwell »

In case someone involved in this reads this thread: the document does not specify which LMDB version you tested. I suggest you run your tests with the `mdb.master` branch, i.e. the work towards a future 1.0, and not the stable 0.9 branch. The answers to several of your interrogations will depend on that: with `mdb.master` you can use the VL32 mode which greatly improves usage on 32 bit platforms, and Windows support is much better.

Regarding NFS: I have recently started testing LMDB on NFS v4 and had no issues so far, but with a single process using the database. AFAIK the warning at http://www.lmdb.tech/doc/ is only for multiple processes using the DB concurrently. I am still not entirely sure there won't be any mmap-related issues, but so far so good.

Regarding "being careful": this is a very important point. The LMDB API does not hold your hand, it lets you do dangerous things which will result in corruption of your database, which you will discover too late. I suggest writing a wrapper around the API to ensure you are using it correctly. (I wish there was a compile flag like LUA_USE_APICHECK [1] for LMDB, which could help detect problems like this, but there isn't.)

[1] https://www.lua.org/manual/5.3/manual.html#4

Re: Design Review: Key-Value Storage

#66

Post by barrkel »

romed wrote: post #57
floatingatoll wrote: post #36

Earlier quoted context omitted.

Transactions let you perform any series of SQL commands with various expectations around data safety and locking guarantees depending on isolation level. Batch writes provide a tiny subset of the full possibilities of transactions. While sufficient in many cases, that cannot be generalized to "LevelDB supports transactions".

That's just your pet definition of transaction. That is not universally accepted.

Atomicity and consistency. The classic case is balance transfer. Batch writes only cope with this to the degree that you can model your computation purely in database primitives that can execute server-side as part of the batch. If your computation involves anything else, such as external data or a computation that can't be modelled, then microbatches aren't good enough: you risk lost updates and inconsistent reads.

Re: Design Review: Key-Value Storage

#67

Post by hyc_symas »

catwell wrote: post #65

In case someone involved in this reads this thread: the document does not specify which LMDB version you tested. I suggest you run your tests with the `mdb.master` branch, i.e. the work towards a future 1.0, and not the stable 0.9 branch. The answers to several of your interrogations will depend on that: with `mdb.master` you can use the VL32 mode which greatly improves usage on 32 bit platforms, and Windows support…

The main reason for that warning about NFS is that people will try it, see that it seems to work, and then get careless and try to use the same DB from two different hosts at once. It's inevitable when you're working on files living on networked filesystems, and it cannot work. NFS doesn't offer any cache coherency guarantees, and the mutexes used for synchronizing writers only work on the host that created them.

Not sure what you're talking about re: the API letting you corrupt your database.

Re: Design Review: Key-Value Storage

#68

Post by hyc_symas »

acqq wrote: post #63
mykmelez wrote: post #23

Earlier quoted context omitted.

According to @hyc in https://monero.stackexchange.com/questions/2606/are-the-lmdb... , "As of v0.10.0, yes the LMDB files are cross-compatible between 32 and 64bit architectures. They have always been cross-compatible between OSs. They are still byte-order dependent but almost everyone uses little-endian CPUs these days so it's not much of an issue."

So the endianness answer is, from your link: "They are still byte-order dependent but almost everyone uses little-endian CPUs these days so it's not much of an issue." Which just means "we solve the problem by ignoring it completely." And what about the limitations on the 32-bit system? Isn't there also needed to use memory space of RAM for the complete size of the database, that's how that database works if I unders…

The endianness answer is "we thought about it, and it's a non-problem 99% of the time, and not worth further thought." You can't use a DB concurrently from multiple hosts, so it's unnecessary to support both endiannesses at once. If you need to copy a DB between different endian hosts, export it with mdb_dump and reimport it with mdb_load on the other machine. Don't be an ass.

There are no limitations on 32bit systems using MDB_VL32 in the 1.0 branch, you're spewing crap without any idea what you're talking about. Whoever's writing on stackoverflow doesn't know what they're talking about either.

> Moreover, there are use-case scenarios where the memory mapped file approach can suffer from problems of unnecessarily reading the page that will anyway be completely overwritten.

That's only true if you use a writable mmap, and the DB is larger than RAM. The default for LMDB is not to use a writable mmap, so this isn't an issue that affects most LMDB users. It is also already documented, so you choose it at your own risk.

Re: Design Review: Key-Value Storage

#69

Post by jnwatson »

acqq wrote: post #63
mykmelez wrote: post #23

Earlier quoted context omitted.

According to @hyc in https://monero.stackexchange.com/questions/2606/are-the-lmdb... , "As of v0.10.0, yes the LMDB files are cross-compatible between 32 and 64bit architectures. They have always been cross-compatible between OSs. They are still byte-order dependent but almost everyone uses little-endian CPUs these days so it's not much of an issue."

So the endianness answer is, from your link: "They are still byte-order dependent but almost everyone uses little-endian CPUs these days so it's not much of an issue." Which just means "we solve the problem by ignoring it completely." And what about the limitations on the 32-bit system? Isn't there also needed to use memory space of RAM for the complete size of the database, that's how that database works if I unders…

The last issue is merely one of performance, not capability.

I’ve successfully used LMDB in 32-bit land, though it takes some effort. I had to scan available virtual memory for the largest contiguous chunk and use that.

Bigger issues are growing the database size and performance in low memory. No transactions can be running to increase the map size, which is usually hard to coordinate.

Also, in low memory situations, LMDB’s performance suffers tremendously. It can be 100x slower, and commits can take seconds. You won’t run into it unless you are really hammering it, and the developer usually won’t notice because they usually have lots of RAM.

Re: Design Review: Key-Value Storage

#70

Post by hyc_symas »

jnwatson wrote: post #69
acqq wrote: post #63

Earlier quoted context omitted.

So the endianness answer is, from your link: "They are still byte-order dependent but almost everyone uses little-endian CPUs these days so it's not much of an issue." Which just means "we solve the problem by ignoring it completely." And what about the limitations on the 32-bit system? Isn't there also needed to use memory space of RAM for the complete size of the database, that's how that database works if I unders…

The last issue is merely one of performance, not capability. I’ve successfully used LMDB in 32-bit land, though it takes some effort. I had to scan available virtual memory for the largest contiguous chunk and use that. Bigger issues are growing the database size and performance in low memory. No transactions can be running to increase the map size, which is usually hard to coordinate. Also, in low memory situations,…

re: 32bit land, You should look into MDB_VL32 in mdb.master as other posters have suggested.

If any single commit takes multiple seconds, you've probably written too much data in a single transaction. Above about 512MB it starts preemptively flushing intermediate pages out, to keep the in-RAM footprint limited. Some of those intermediate pages will need to be read back in if your transactions are hitting all over the DB. This is where the main slowdown comes.

Post reply on HN