Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Design Review: Key-Value Storage

mozilla.github.io

Post reply on HN
Print view View original

51–60 of 90 posts

Re: Design Review: Key-Value Storage

#51

Post by hyc_symas »

jimis wrote: post #43
hyc_symas wrote: post #42

Earlier quoted context omitted.

We have only ever compared LMDB in synchronous mode to other DBs in synchronous mode, and LMDB in asynch mode to other DBs in asynch mode. Come on, that's too obvious. And LMDB beats the crap out of SQLite, in any mode. http://www.lmdb.tech/bench/microbench/ Replacing SQLite's Btree engine with LMDB makes the SQLite footprint smaller, faster, and more reliable too. https://github.com/LMDB/sqlightning

It's not that simple. What you mean by saying "(a)synchronous mode" is very different from database to database. See my above comment. Is SQLite synchronous or asynchronous if you configure it as journal_mode=WAL and PRAGMA synchronous=NORMAL? For my purposes as an application developer, I care about comparing databases operating in safe mode i.e. a system crash should never cause total data loss. According to my exp…

SQLite's safe mode is not comparable to LMDB's; SQLite is vulnerable to silent data loss in a crash.

https://wisdom.cs.wisc.edu/workshops/spring-14/talks/Thanu.p...

LMDB is not.

Re: Design Review: Key-Value Storage

#52

Post by AboutTheWhisles »

It really sounds like they should take a look at this:

https://github.com/LiveAsynchronousVisualizedArchitecture/si...

"One appealing aspect of LMDB is its relative ease of use from multiple processes, above and beyond its basic capabilities as yet-another-fast-key-value-store."

simdb is only lock free and thread safe. While LMDB is benchmarked at around 10k writes, this should be able to do millions of mixed reads and writes with 4 modern cores. LMDB seems to use a separate lock file to sync multiple threads/processes. The only catch here is that the keys aren't sorted, which doesn't seem to be a requirement of theirs.

Re: Design Review: Key-Value Storage

#53

Post by mbreese »

adobeeee wrote: post #50

I have a layman question if somebody could please answer. I have never in my entire life seen databases fail. But db failures and issues seem to be brought up all the time. Now I understand that part if this maybe the cost function associated with them. But I'm sure there's also something that I have no clue about. So my questions are: 1) what kind of problems do databases actually face. 2) what kind of scenarios cre…

The easiest scenario to imagine is a hardware failure or power outage. The database was in the middle of doing something, and then was prevented by a hard drive dying or the lights going out. One way to test such a thing is to literally unplug the computer to see how it handles the failure.

So, let's say you have a client/server application... the client is telling the server (database) to write some records to the database. In the middle of the write, you pull the plug. Some questions you'd want to know: what does the database look like when it restarts? Can we read it? What is the current state? Did any of the new data get written? What does the client think was written? If there was an uncommitted database transaction, was the database left unaltered?

It's just as important to test the client in these scenarios. While the server may have crashed, what does the client think happened? Was it waiting for an ACK or "OK" message? Did it get the message? If the update failed, what does the client do in that situation?

Things can get even more complicated if you're thinking of replication across different servers. If one of the servers fails, how does the replication work? Do sessions fail over to other servers? How many servers are required? If there was a corrupted record, did it propagate or was it scrubbed?

Re: Design Review: Key-Value Storage

#55

Post by jimis »

hyc_symas wrote: post #51
jimis wrote: post #43

Earlier quoted context omitted.

It's not that simple. What you mean by saying "(a)synchronous mode" is very different from database to database. See my above comment. Is SQLite synchronous or asynchronous if you configure it as journal_mode=WAL and PRAGMA synchronous=NORMAL? For my purposes as an application developer, I care about comparing databases operating in safe mode i.e. a system crash should never cause total data loss. According to my exp…

SQLite's safe mode is not comparable to LMDB's; SQLite is vulnerable to silent data loss in a crash. https://wisdom.cs.wisc.edu/workshops/spring-14/talks/Thanu.p... LMDB is not.

Isn't this from the paper "all filesystems are not created equal"? [1] If you search the tables for "sqlite-wal" you will see that it shows zero vulnerabilities.

[1] https://www.usenix.org/system/files/conference/osdi14/osdi14...

Re: Design Review: Key-Value Storage

#56

Post by positr0n »

adobeeee wrote: post #50

I have a layman question if somebody could please answer. I have never in my entire life seen databases fail. But db failures and issues seem to be brought up all the time. Now I understand that part if this maybe the cost function associated with them. But I'm sure there's also something that I have no clue about. So my questions are: 1) what kind of problems do databases actually face. 2) what kind of scenarios cre…

You'd be surprised by how often I've seen a database fail in prod simply because it ran out of disk space. In both cases monitoring software was running but misconfigured.

Re: Design Review: Key-Value Storage

#57

Post by romed »

floatingatoll wrote: post #36
shekispeaks wrote: post #17

This doc says LevelDB has no transacion, thats not true, they have batch writes and LEVEL-DB is not implemented in Go. Its implemented in CPP

Transactions let you perform any series of SQL commands with various expectations around data safety and locking guarantees depending on isolation level. Batch writes provide a tiny subset of the full possibilities of transactions. While sufficient in many cases, that cannot be generalized to "LevelDB supports transactions".

That's just your pet definition of transaction. That is not universally accepted.

Re: Design Review: Key-Value Storage

#59

Post by dang »

coleifer wrote: post #33
mrjn wrote: post #13

Earlier quoted context omitted.

Author of Badger here. Our design of separating keys and values has gotten us incredibly fast writes, while still keeping the read latencies neck-to-neck against B+ trees. Worth checking out: https://github.com/dgraph-io/badger

I mean...whatever, sure I'll take your word for it. I don't recall anyone talking about badgers though.

Please don't be a jerk in comments here.

Edit: sadly, it looks like you've been posting quite a few uncivil comments to HN. We ban accounts that do that, so would you please review https://news.ycombinator.com/newsguidelines.html and follow the rules from now on?

The idea here is: if you have a substantive point to make, make it respectfully and thoughtfully; if you don't, please don't comment until you do.

Re: Design Review: Key-Value Storage

#60

Post by adobeeee »

mbreese wrote: post #53
adobeeee wrote: post #50

I have a layman question if somebody could please answer. I have never in my entire life seen databases fail. But db failures and issues seem to be brought up all the time. Now I understand that part if this maybe the cost function associated with them. But I'm sure there's also something that I have no clue about. So my questions are: 1) what kind of problems do databases actually face. 2) what kind of scenarios cre…

The easiest scenario to imagine is a hardware failure or power outage. The database was in the middle of doing something, and then was prevented by a hard drive dying or the lights going out. One way to test such a thing is to literally unplug the computer to see how it handles the failure. So, let's say you have a client/server application... the client is telling the server (database) to write some records to the d…

Thank you for explaining so well and clearly!

To you and others, are there any other scenarios too that happen in production?

Post reply on HN