Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Design Review: Key-Value Storage

mozilla.github.io

Post reply on HN
Print view View original

31–40 of 90 posts

Re: Design Review: Key-Value Storage

#31

Post by drzaiusx11 »

drzaiusx11 wrote: post #30
Mossop wrote: post #26

Earlier quoted context omitted.

There is no intent to standardize this or expose it to the web. This is purely for use internal to Firefox. Of course that may end up meaning that it is used as the internal storage for some web feature.

The first sentence on that page says "We propose the standardization of a simple key-value storage...usable from JS, Java, Rust, Swift, and C++" I assumed this meant an API callable from webasm/js. Did I miss something?

Ah, found my error:

"Not-yet or never goals for this proposal are:

Standardization via a standards body as a web API."

So this is "internal" stuff I guess.

Re: Design Review: Key-Value Storage

#33

Post by coleifer »

mrjn wrote: post #13
coleifer wrote: post #7

Earlier quoted context omitted.

Also typically more complicated and require a separate compaction process. They're good for writing lots of data, but not so great for random reads.

Author of Badger here. Our design of separating keys and values has gotten us incredibly fast writes, while still keeping the read latencies neck-to-neck against B+ trees. Worth checking out: https://github.com/dgraph-io/badger

I mean...whatever, sure I'll take your word for it. I don't recall anyone talking about badgers though.

Re: Design Review: Key-Value Storage

#35

Post by malkia »

jimis wrote: post #28

The page writes that LMDB "is exceptionally fast for our kinds of load", and then links to an in-memory microbenchmark: http://www.lmdb.tech/bench/inmem/ Aren't they interested in persistence of the key-value data? In my experience, once data is persisted to disk or SSD, LMDB is way slower from alternatives because it needs to operate in synchronous mode to avoid corruption (effectively flushing to disk after after e…

Concidentally, I'm in the middle of testing out LMDB as replacement for SQLite - we've been running fine for a year with journal_mode=WAL, but synchronos=NORMAL without issues, and just added last week LMDB, but with MDB_NOSYNC... hmm... so far our real testing showed (for our use case) almost same results, except some 95-99% percentiles where SQLite goes much slower (but due to other factors). In synthetic benchmarking (suiting our needs) the difference is miniscule..

Re: Design Review: Key-Value Storage

#36

Post by floatingatoll »

shekispeaks wrote: post #17

This doc says LevelDB has no transacion, thats not true, they have batch writes and LEVEL-DB is not implemented in Go. Its implemented in CPP

Transactions let you perform any series of SQL commands with various expectations around data safety and locking guarantees depending on isolation level.

Batch writes provide a tiny subset of the full possibilities of transactions. While sufficient in many cases, that cannot be generalized to "LevelDB supports transactions".

Re: Design Review: Key-Value Storage

#37

Post by jimis »

malkia wrote: post #35
jimis wrote: post #28

The page writes that LMDB "is exceptionally fast for our kinds of load", and then links to an in-memory microbenchmark: http://www.lmdb.tech/bench/inmem/ Aren't they interested in persistence of the key-value data? In my experience, once data is persisted to disk or SSD, LMDB is way slower from alternatives because it needs to operate in synchronous mode to avoid corruption (effectively flushing to disk after after e…

Concidentally, I'm in the middle of testing out LMDB as replacement for SQLite - we've been running fine for a year with journal_mode=WAL, but synchronos=NORMAL without issues, and just added last week LMDB, but with MDB_NOSYNC... hmm... so far our real testing showed (for our use case) almost same results, except some 95-99% percentiles where SQLite goes much slower (but due to other factors). In synthetic benchmark…

If you are using MDB_NOSYNC in production, keep in mind that the trade-off for the boost in performance is that a system crash can lead to an unreadable database.

Re: Design Review: Key-Value Storage

#38

Post by malkia »

jimis wrote: post #37
malkia wrote: post #35

Earlier quoted context omitted.

Concidentally, I'm in the middle of testing out LMDB as replacement for SQLite - we've been running fine for a year with journal_mode=WAL, but synchronos=NORMAL without issues, and just added last week LMDB, but with MDB_NOSYNC... hmm... so far our real testing showed (for our use case) almost same results, except some 95-99% percentiles where SQLite goes much slower (but due to other factors). In synthetic benchmark…

If you are using MDB_NOSYNC in production, keep in mind that the trade-off for the boost in performance is that a system crash can lead to an unreadable database.

Yup, we had SQLITE in the same fashion (synchrnous=OFF, and other non-safe settings, no WAL also) and had occasional crashes, then someone had to manually delete the database (in our case it's for a desktop app, a tool, so no server-like requirements, yet we want it to be stable).

Would remove the MDB_NOSYNC then, and see how it goes...

Re: Design Review: Key-Value Storage

#39

Post by CardenB »

jimis wrote: post #28

The page writes that LMDB "is exceptionally fast for our kinds of load", and then links to an in-memory microbenchmark: http://www.lmdb.tech/bench/inmem/ Aren't they interested in persistence of the key-value data? In my experience, once data is persisted to disk or SSD, LMDB is way slower from alternatives because it needs to operate in synchronous mode to avoid corruption (effectively flushing to disk after after e…

How does this compare to sstable?

Re: Design Review: Key-Value Storage

#40

Post by hyc_symas »

mamcx wrote: post #9

I see their have some questions (like how good is the windows and android support) that are not answered? or only internally? I think will be good to see what them found.

Windows and Android are fully supported, have been for years. As are iOS and MacOSX and all the BSDs.
Post reply on HN