Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Design Review: Key-Value Storage

mozilla.github.io

Post reply on HN
Print view View original

71–80 of 90 posts

Re: Design Review: Key-Value Storage

#71

Post by coleifer »

dang wrote: post #59
coleifer wrote: post #33

Earlier quoted context omitted.

I mean...whatever, sure I'll take your word for it. I don't recall anyone talking about badgers though.

Please don't be a jerk in comments here. Edit: sadly, it looks like you've been posting quite a few uncivil comments to HN. We ban accounts that do that, so would you please review https://news.ycombinator.com/newsguidelines.html and follow the rules from now on? The idea here is: if you have a substantive point to make, make it respectfully and thoughtfully; if you don't, please don't comment until you do.

Hey, I know you care a lot about this forum. Sometimes its easy to forget what this place means to some folks and just do a drive-by shitpost. I hear you.

Re: Design Review: Key-Value Storage

#72

Post by marcosdumay »

adobeeee wrote: post #60
mbreese wrote: post #53

Earlier quoted context omitted.

The easiest scenario to imagine is a hardware failure or power outage. The database was in the middle of doing something, and then was prevented by a hard drive dying or the lights going out. One way to test such a thing is to literally unplug the computer to see how it handles the failure. So, let's say you have a client/server application... the client is telling the server (database) to write some records to the d…

Thank you for explaining so well and clearly! To you and others, are there any other scenarios too that happen in production?

Some things I have seen in production:

The disk become inoperative during a write, this can be either silent or writes start to return errors. Again, how does the database look like after the problem is solved.

A large operation exceeds the capacity of the server to deal with intermediary state. It runs out of memory, disk, or in some not great DBs it loses control of some locks and gets deadlocked. Can it recover with only the partial log data?

Disks lie about data being written, what happens if one of the problems happen between the disk saying the data was written and it actually getting written?

And, of course, when you move beyond a single server things get way more complex.

Re: Design Review: Key-Value Storage

#74

Post by Twirrim »

erpellan wrote: post #64
Twirrim wrote: post #61

Earlier quoted context omitted.

I would love to see an example of someone actually knowledgeable about databases who has a different definition for transaction.

Atomicity, consistency, isolation, durability. https://en.wikipedia.org/wiki/ACID_(computer_science) But that's just like, the industry's opinion, man.

Maybe I completely misunderstood the parents description of a transaction, but that was exactly what they were saying.

Re: Design Review: Key-Value Storage

#76

Post by Mossop »

drzaiusx11 wrote: post #31
drzaiusx11 wrote: post #30

Earlier quoted context omitted.

The first sentence on that page says "We propose the standardization of a simple key-value storage...usable from JS, Java, Rust, Swift, and C++" I assumed this meant an API callable from webasm/js. Did I miss something?

Ah, found my error: "Not-yet or never goals for this proposal are: Standardization via a standards body as a web API." So this is "internal" stuff I guess.

Poor wording perhaps. There are a bunch of places in Firefox that use key-value stores, this proposes standardising on one type of store.

Re: Design Review: Key-Value Storage

#77

Post by mykmelez »

acqq wrote: post #63
mykmelez wrote: post #23

Earlier quoted context omitted.

According to @hyc in https://monero.stackexchange.com/questions/2606/are-the-lmdb... , "As of v0.10.0, yes the LMDB files are cross-compatible between 32 and 64bit architectures. They have always been cross-compatible between OSs. They are still byte-order dependent but almost everyone uses little-endian CPUs these days so it's not much of an issue."

So the endianness answer is, from your link: "They are still byte-order dependent but almost everyone uses little-endian CPUs these days so it's not much of an issue." Which just means "we solve the problem by ignoring it completely." And what about the limitations on the 32-bit system? Isn't there also needed to use memory space of RAM for the complete size of the database, that's how that database works if I unders…

> Again, it seems that for LMDB is again "solving" the problem by ignoring it. Which is OK if it fits your use case... But shouldn't Firefox work properly on 32-bit systems? Or did they completely decide that they don't want to target any 32-bit platform anymore?

Yes and no, respectively. Yes, Firefox should work properly on 32-bit systems; and no, we didn't completely decide that we don't want to target 32-bit platforms anymore.

Rather, Firefox plans to use LMDB where it fits one of its many use cases for persistent storage (and not use it where it doesn't).

The StackOverflow thread you referenced describes a blockchain program that expects to use "a few GB of lmdb diskspace." Whereas Firefox's use cases for LMDB are sized in the range of a few KiB to a few MiB. Firefox has no plans to start storing a blockchain.

Claimer: I'm the engineer integrating LMDB into Firefox.

Re: Design Review: Key-Value Storage

#78

Post by justin66 »

hyc_symas wrote: post #67
catwell wrote: post #65

In case someone involved in this reads this thread: the document does not specify which LMDB version you tested. I suggest you run your tests with the `mdb.master` branch, i.e. the work towards a future 1.0, and not the stable 0.9 branch. The answers to several of your interrogations will depend on that: with `mdb.master` you can use the VL32 mode which greatly improves usage on 32 bit platforms, and Windows support…

The main reason for that warning about NFS is that people will try it, see that it seems to work, and then get careless and try to use the same DB from two different hosts at once. It's inevitable when you're working on files living on networked filesystems, and it cannot work. NFS doesn't offer any cache coherency guarantees, and the mutexes used for synchronizing writers only work on the host that created them. Not…

If we're using lmdb for something that's largely or entirely read only, how viable is using it on NFS with (shudder) lock files or something like that?

Re: Design Review: Key-Value Storage

#79

Post by iplusplus »

coleifer wrote: post #71
dang wrote: post #59

Earlier quoted context omitted.

Please don't be a jerk in comments here. Edit: sadly, it looks like you've been posting quite a few uncivil comments to HN. We ban accounts that do that, so would you please review https://news.ycombinator.com/newsguidelines.html and follow the rules from now on? The idea here is: if you have a substantive point to make, make it respectfully and thoughtfully; if you don't, please don't comment until you do.

Hey, I know you care a lot about this forum. Sometimes its easy to forget what this place means to some folks and just do a drive-by shitpost. I hear you.

In fact he cares so much about this forum that he won't let 12-year members participate fairly on it, and claims that mocking him when he acts like a dictator is "abusive to the community." He considers himself the community.

Don't take this place too seriously. You may find that 12 years later, all of your hard work in writing quality comments is for nothing, just because of the whims and moods of Pilate.

If you try to ask him "Why?" he will not explain the reasons (unless they are very clear-cut and obvious) and instead falls back on demanding respect and unquestioning obedience. It's a shame, because pg's reign was at least tactful.

The worst part is, if you confront him about any of this, he will claim it's not true and ban you permanently if you persist. (Well, if you persist publicly, at least.)

It's a good idea to suck up, though. If you do want to participate earnestly and in the long-term, I suggest getting used to that.

Re: Design Review: Key-Value Storage

#80

Post by tfha »

adobeeee wrote: post #50

I have a layman question if somebody could please answer. I have never in my entire life seen databases fail. But db failures and issues seem to be brought up all the time. Now I understand that part if this maybe the cost function associated with them. But I'm sure there's also something that I have no clue about. So my questions are: 1) what kind of problems do databases actually face. 2) what kind of scenarios cre…

We ship a consumer application with databases in it (multiple). Even the ACID ones fail all the time, maybe 1-3% of our userbase has had a corruption at some point.
Post reply on HN