Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Design Review: Key-Value Storage

mozilla.github.io

Post reply on HN
Print view View original

21–30 of 90 posts

Re: Design Review: Key-Value Storage

#21

Post by pvg »

majewsky wrote: post #19
jcranmer wrote: post #8

Earlier quoted context omitted.

Mork actually sucked even as a key-value store. It's only decent if your requirements are a) only lookup on a fixed, autoincrement integer ID, b) the only operation you're likely to do is load an entire record or store an entire record at once, and c) parallelism is not in your vocabulary. Disclaimer: I'm one of the last people to make functional changes to mork.

That's not a "Disclaimer". It's actually the polar opposite... a "Claimer"? ... I think "Source" is the best word to use there.

Welcome to the war, majewsky.

https://news.ycombinator.com/item?id=17607457#17618288

Re: Design Review: Key-Value Storage

#22

Post by derefr »

shekispeaks wrote: post #17

This doc says LevelDB has no transacion, thats not true, they have batch writes and LEVEL-DB is not implemented in Go. Its implemented in CPP

> LEVEL-DB is not implemented in Go. Its implemented in CPP

There is more than one implementation of LevelDB. This one (https://github.com/syndtr/goleveldb), in Go, is used in major projects such as https://github.com/ethereum/go-ethereum.

Re: Design Review: Key-Value Storage

#23

Post by mykmelez »

acqq wrote: post #18

I thought that the issues like this would be blocking: https://stackoverflow.com/questions/44407659/how-to-force-64... Who knows how it should work on 32-bit systems? And isn't endianess also a problem? And doesn't SQLite solve both problems by default? And isn't also possible to configure SQLite to be very fast, if one know what he's doing? Btw I'd expected that the "notes here" https://docs.google.com/document/d/1b…

According to @hyc in https://monero.stackexchange.com/questions/2606/are-the-lmdb..., "As of v0.10.0, yes the LMDB files are cross-compatible between 32 and 64bit architectures. They have always been cross-compatible between OSs. They are still byte-order dependent but almost everyone uses little-endian CPUs these days so it's not much of an issue."

Re: Design Review: Key-Value Storage

#24

Post by drzaiusx11 »

Hopeful this goes better than WebSQL. WebSQL ran into standardization problems because they started with an existing implementation (SQLite) instead of a spec as their base implementation--meaning anyone making a greenfield implementation would need to implement all the quirks of SQLite to be compatible with other implementations. Sadly it died as a standard and we were stuck with either local storage or indexdb.

Re: Design Review: Key-Value Storage

#25

Post by majewsky »

pvg wrote: post #21
majewsky wrote: post #19

Earlier quoted context omitted.

That's not a "Disclaimer". It's actually the polar opposite... a "Claimer"? ... I think "Source" is the best word to use there.

Welcome to the war, majewsky. https://news.ycombinator.com/item?id=17607457#17618288

Hah! :) I've been meaning to call out other misuses of "Disclaimer" vs. "Disclosure" before, but I usually don't because it's just tedious.

Re: Design Review: Key-Value Storage

#26

Post by Mossop »

drzaiusx11 wrote: post #24

Hopeful this goes better than WebSQL. WebSQL ran into standardization problems because they started with an existing implementation (SQLite) instead of a spec as their base implementation--meaning anyone making a greenfield implementation would need to implement all the quirks of SQLite to be compatible with other implementations. Sadly it died as a standard and we were stuck with either local storage or indexdb.

There is no intent to standardize this or expose it to the web. This is purely for use internal to Firefox. Of course that may end up meaning that it is used as the internal storage for some web feature.

Re: Design Review: Key-Value Storage

#27

Post by digikata »

mrjn wrote: post #13
coleifer wrote: post #7

Earlier quoted context omitted.

Also typically more complicated and require a separate compaction process. They're good for writing lots of data, but not so great for random reads.

Author of Badger here. Our design of separating keys and values has gotten us incredibly fast writes, while still keeping the read latencies neck-to-neck against B+ trees. Worth checking out: https://github.com/dgraph-io/badger

Badger looks nicely done! Did you end up needing to change much in implementing Badger from what was described in the WiscKey paper?

Re: Design Review: Key-Value Storage

#28

Post by jimis »

The page writes that LMDB "is exceptionally fast for our kinds of load", and then links to an in-memory microbenchmark: http://www.lmdb.tech/bench/inmem/

Aren't they interested in persistence of the key-value data? In my experience, once data is persisted to disk or SSD, LMDB is way slower from alternatives because it needs to operate in synchronous mode to avoid corruption (effectively flushing to disk after after every transaction committed). If operated in the non-default MDB_NOSYNC mode (which is the mode chosen in the above benchmarks), then there is a high probability to be left with an unreadable database file after a crash, thus losing all your data.

It is not fair to compare with other databases in sync mode, since they might operate safe but faster in async mode. For example sqlite with PRAGMA journal_mode=WAL and PRAGMA synchronous=NORMAL can operate in semi-asynchronous mode (fsync()ing sporadically) without fear of corruption in case of crash, because it keeps a WAL journal and is able to properly roll-back after a crash. This should be much faster than LMDB's default-and-safe synchronous mode, that msync()s on every value written.

Re: Design Review: Key-Value Storage

#29

Post by espeed »

majewsky wrote: post #19
jcranmer wrote: post #8

Earlier quoted context omitted.

Mork actually sucked even as a key-value store. It's only decent if your requirements are a) only lookup on a fixed, autoincrement integer ID, b) the only operation you're likely to do is load an entire record or store an entire record at once, and c) parallelism is not in your vocabulary. Disclaimer: I'm one of the last people to make functional changes to mork.

That's not a "Disclaimer". It's actually the polar opposite... a "Claimer"? ... I think "Source" is the best word to use there.

NB: NB works for me.

Re: Design Review: Key-Value Storage

#30

Post by drzaiusx11 »

Mossop wrote: post #26
drzaiusx11 wrote: post #24

Hopeful this goes better than WebSQL. WebSQL ran into standardization problems because they started with an existing implementation (SQLite) instead of a spec as their base implementation--meaning anyone making a greenfield implementation would need to implement all the quirks of SQLite to be compatible with other implementations. Sadly it died as a standard and we were stuck with either local storage or indexdb.

There is no intent to standardize this or expose it to the web. This is purely for use internal to Firefox. Of course that may end up meaning that it is used as the internal storage for some web feature.

The first sentence on that page says "We propose the standardization of a simple key-value storage...usable from JS, Java, Rust, Swift, and C++"

I assumed this meant an API callable from webasm/js. Did I miss something?

Post reply on HN