Live data from Hacker News

Up to date Rust bindings for LMDB

crates.io

11–15 of 15 posts

Re: Up to date Rust bindings for LMDB

#11
post #9

What kinds of use cases do you all use LMDB or other key-value stores for? I can somehow never really find a good one. Almost all databases, no matter how trivial, will sooner rather than later need more features so I always reach for SQLite instead. Still, I'd really like to make use of it. But for what? Caching?

I've never used LMDB directly myself, but it's used as the backing store for stuff like some LDAP servers. You get a high performance low-level layer and anything extra you build on top of it yourself

Re: Up to date Rust bindings for LMDB

#12
post #9

What kinds of use cases do you all use LMDB or other key-value stores for? I can somehow never really find a good one. Almost all databases, no matter how trivial, will sooner rather than later need more features so I always reach for SQLite instead. Still, I'd really like to make use of it. But for what? Caching?

It was originally designed for LDAP, so there is one example. Using a relational system almost certainly means you're using key-value structures indirectly. That's how table indexes are often implemented: a map from some key to a row coordinate (or what have you.) LMDB has been used as a backend for SQLite.

Key-value stores are primitives. Sometimes your use case needs nothing more elaborate than this primitive. Sometimes you have no choice but to limit yourself to that primitive because there are no resources for anything more complex. Often you're using it as a component of a larger storage scheme.

Mail systems are an example where you're forever looking up senders and/or recipients for some policy. Recently I've dealt with GeoIP stuff, where you're constantly resolving network addresses against a huge map. Bloom filters are a great use of a key-value store for large data sets.

Understand that often there is a lot of custom code around queries and other operations that use key-value stores. You might, for example, have some tuple that can map to multiple values in a sparse map. In that case you can iterate over variants of the tuple (selectively remove, normalize or otherwise alter parts of it,) and make multiple queries according to some precedence rule until you get a hit (or not.) There are many such tricks that will greatly extend the utility of a map.

LMDB in particular is worth understanding, even if you never use it. It has an extremely simple design that essentially makes the OS page cache into an ACID database with concurrent readers that are not blocked by writes or require coordination, and has no "transaction log." Pretty astonishing given that it boils down to about 10K LOC.

Re: Up to date Rust bindings for LMDB

#13

not sure why all the existing crates have gone so far out of date. i feel like generating bindings isn't that hard if you've done it before but can be a blocker for people who haven't so maybe this will help. some of the existing crates had the -sys suffix but it didn't seem like they were actually checking for it on the system (i'm not sure that LMDB is installed by default on most systems anyway?) so opted to not u…

I always saw the "-sys" suffix as "raw, unsafe bindings", not pulling it from the system. And non-sys is your safe Rust abstraction.

Re: Up to date Rust bindings for LMDB

#14
post #9

What kinds of use cases do you all use LMDB or other key-value stores for? I can somehow never really find a good one. Almost all databases, no matter how trivial, will sooner rather than later need more features so I always reach for SQLite instead. Still, I'd really like to make use of it. But for what? Caching?

LMDB is kinda unique in KV segment. It's fast, gives access to zerocopy values, full ACID. I use it for inverted index where SQLite quite sucks.

The only issue with LMDB is a size reservation for a db file.

Re: Up to date Rust bindings for LMDB

#15
post #8

not sure why all the existing crates have gone so far out of date. i feel like generating bindings isn't that hard if you've done it before but can be a blocker for people who haven't so maybe this will help. some of the existing crates had the -sys suffix but it didn't seem like they were actually checking for it on the system (i'm not sure that LMDB is installed by default on most systems anyway?) so opted to not u…

LMDB Python maintainer here. 0.9.70 is not really the latest release. It isn't binary compatible with official "releases" though it has sneaked out in a few bindings. Latest is 0.9.31.

Thank you for your work! I'm a user of py-lmdb for 8 years.
Post reply on HN