Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

WhiteDB – Lightweight NoSQL database written in C, operating in main memory

whitedb.org

Post reply on HN
Print view View original

41–50 of 83 posts

Re: WhiteDB – Lightweight NoSQL database written in C, operating in main memory

#41

Post by danjayh »

tammet wrote: post #17

One of the authors here. A few answers quickly. It does write to disk: you can either dump memory or write all changes to log (turn it on/off yourself). Sure it has a global read/write lock, with several locking strategies to select from (task-fair atomic spinlock queue or a reader-preference or a writer-preference spinlock). It is definitely meant to be a simple library. We strived to document it carefully to make u…

Is there information available anywhere on the required resources? Specifically:

1) How much space does the compiled code require? Can conditional compilation be used to omit unused features?

2) What is the overhead for the various data structures?

I'm thinking that it might be interesting to use this on very limited environments (PIC microcontrollers for example) where every byte matters.

Re: WhiteDB – Lightweight NoSQL database written in C, operating in main memory

#42

Post by tammet »

danjayh wrote: post #41
tammet wrote: post #17

One of the authors here. A few answers quickly. It does write to disk: you can either dump memory or write all changes to log (turn it on/off yourself). Sure it has a global read/write lock, with several locking strategies to select from (task-fair atomic spinlock queue or a reader-preference or a writer-preference spinlock). It is definitely meant to be a simple library. We strived to document it carefully to make u…

Is there information available anywhere on the required resources? Specifically: 1) How much space does the compiled code require? Can conditional compilation be used to omit unused features? 2) What is the overhead for the various data structures? I'm thinking that it might be interesting to use this on very limited environments (PIC microcontrollers for example) where every byte matters.

Conditional compilation can be quite certainly used. The best way to find out the space requirements is to try out some of the examples provided. I cannot give the overhead exactly, but it can be read from the source with not too much effort. Send an email to tanel.tammet at gmail.com if you need help with that. In broad terms, we have been very careful with using memory, both for the reasons you state and the reason of getting more bang from the cache.

Re: WhiteDB – Lightweight NoSQL database written in C, operating in main memory

#43

Post by sakai »

Does anyone mind sharing what they see as the advantage of this vis-a-vis a Redis, LevelDB, Mongo, etc. (realizing those are all very different)? Is it principally read throughput, write throughput, interfacing, space efficiency, scalability to very large datasets, or something else?

I ask as friends and I are doing research into very space-efficient (read: probabilistic) key-value stores. We have a few scientific use cases, but I'm curious if others would find the ability to scale to very large key- and value spaces (~1e9+ key-value pairs) in a space efficient way practically useful. Or, if interest is principally academic.

Apologies for the (pseudo-)hijack.

Ps. Looks very interesting, I have it on my to-dos to install and check it out more deeply.

Re: WhiteDB – Lightweight NoSQL database written in C, operating in main memory

#44

Post by otterley »

tammet wrote: post #17

One of the authors here. A few answers quickly. It does write to disk: you can either dump memory or write all changes to log (turn it on/off yourself). Sure it has a global read/write lock, with several locking strategies to select from (task-fair atomic spinlock queue or a reader-preference or a writer-preference spinlock). It is definitely meant to be a simple library. We strived to document it carefully to make u…

Why is shared memory better than mmap(2) here? With the latter, you get persistence for free.

Re: WhiteDB – Lightweight NoSQL database written in C, operating in main memory

#46

Post by lttlrck »

throwaway420 wrote: post #8

Is it accurate to describe this as a faster Redis that doesn't have all of the useful data structures like lists, sets, etc?

Probably. Though it would be inaccurate to say it is in any way equivalent to Redis.

Re: WhiteDB – Lightweight NoSQL database written in C, operating in main memory

#47

Post by ateeqs »

But why exactly is an in-memory NoSQL database necessary? This is a "non-sequitur." The fundamental purpose to using NoSQL databases is when your data spans multi-terabytes (where "multi" > 4) and you need replication and slicing. Main-memory NoSQL database seems unnecessary, doesn't it?

Also, when you say "main memory" NoSQL database, are you saying "never store in page file" but always reside/lock in main memory? If it will go to the page file, it's not really main-memory, is it?

Re: WhiteDB – Lightweight NoSQL database written in C, operating in main memory

#49

Post by malkia »

jonny_eh wrote: post #15

Does it write to disk ever? I could just assume that it does because it's a DB, and would be nearly worthless if it didn't, but it keeps talking about being only in memory.

It's using memory mapped files, so that's why I guess. Not sure if these are portable between architectures.

Re: WhiteDB – Lightweight NoSQL database written in C, operating in main memory

#50

Post by aryastark »

ateeqs wrote: post #47

But why exactly is an in-memory NoSQL database necessary? This is a "non-sequitur." The fundamental purpose to using NoSQL databases is when your data spans multi-terabytes (where "multi" > 4) and you need replication and slicing. Main-memory NoSQL database seems unnecessary, doesn't it? Also, when you say "main memory" NoSQL database, are you saying "never store in page file" but always reside/lock in main memory? I…

Pretty much what I was trying to figure out. I can't see a use case here, since we already have things like Berkeley DB and LevelDB. But main memory? Your use case would have to be such that your entire dataset can fit in memory and for some reason you need extreme performance from that dataset, and from multiple processes running on a single machine. I just don't see it. Is the caching performance of other databases so horrible that this is really necessary?
Post reply on HN