Badger – A fast key-value store written natively in Go

> From what I've heard of lawyers in the Free Software world, they most applauf AGPLv3 for its clarity and precision.

You're sure they were talking AGPLv3 and not [L]GPLv3?

The definitions of what constitutes an interactive program is quite vague (sect 0 and 13). Let's say you have a database server under AGPL (mongo, or say citus). Clearly they support interactive access in some form, but from the perspective of user of an application using said database access is not interactive, nor is it clear how the database could provide such an interactive notice. Various vendors addressed that issue with clarifying notes about their understanding, but that definitely increases doubts of possible users, including their lawyers.

I'm skeptical that that would enable reuse of existing C code.

If the workflow requires you to modify or annotate the C code, that's a lot of work (and risky) and you might as well just rewrite it in Go.

If the workflow is totally automatic, so you can just use off-the-shelf C code, that's great! But in that case it's effectively just a C compiler, the "Go--" bit seems like a red herring.

It reminds me a bit of "C--", Simon Peyton Jones' suggestion for a low-level target for Haskell and similar languages. It would remove some C functionality that language runtimes don't really need, and add some extra low-level stuff like register globals and tail calls. I don't think that got much traction, but it may have influenced the design of LLVM.

Go has no native support for aio. Based on this thread, Goroutines seem to do the same thing, via epolls. https://groups.google.com/forum/#!topic/golang-nuts/AQ8JOHxm...

I think the best bet is to build a fio equivalent in Go (shouldn't take more than a couple of hours), and see if it can achieve the same throughput as fio itself. That can help figure out how slow is Go compared to using libaio directly via C.

I threw a simple smoke test on it: walk a directory tree and store each regular file in the tree in the data storage thing. The directory tree wasn't anything radical - just 112MB, 260 keys/values, biggest one was 10MB. Then close the storage thing, open it again and see if we can retrieve the contents. No concurrency, no crash recovery, nothing. Sure, maybe not exactly a typical workload, but let's see how things behave at the extremes first. They do talk about big keys in the blog post.

First impression. Easy to write the code.

Second impression. Fast as hell.

Third impression. Loses data. The last file in the set is consistently corrupted no matter how many times I try. I also tried on different directories. Exact same result everywhere. Last key/value written is truncated.

Hmm. A data storage thing that fails to store data is maybe not exactly invoking feelings of trust, which should be the primary feelings you have about your data storage thing.

Let's just throw it on something bigger. My /usr/lib. It's just 1.4GB. Shouldn't be too hard.

    [ 6546.783474] Killed process 4895 (main) total-vm:497120kB, anon-rss:368384kB, file-rss:0kB, shmem-rss:0kB

Hmm. Do they just store everything in memory forever? Sure, this VM has stupidly little memory. But that's why we have disk. So that we don't have to keep everything in memory. But good. Now we get to test the crash recovery mechanism.

    [ 6823.635967] Killed process 4919 (main) total-vm:677964kB, anon-rss:367980kB, file-rss:0kB, shmem-rss:0kB

Right. I guess there are reasons why experience taught me to let others be the first to use new data storage technology for a few years.

In case someone wonders the test code is here: https://gist.github.com/art4711/9d781b8cf1f9f36df73a9c5c0403...

It may be all wrong, but considering that the "docs" directory contains 6MB of stuff just to display "hello world" twice, I had to write it based on what godoc said.

> We have tried with BoltDB. Its performance is really bad.

BoltDB author here. I agree with you that Badger would beat Bolt in many performance benchmarks but it's an apples and oranges comparison. LSM tree key/value stores are write-optimized and generally lack transactional support. BoltDB is read-optimized and supports ACID transactions with serializable isolation. Transactions come with a cost but they're really important for most applications.

Regarding benchmarks, LSMs typically excel in random and sequential write performance and do OK with random read performance. LSMs tend to be terrible for sequential scans since levels have to be merged at query time. B+trees are usually terrible at random write performance but can do well with sequential writes that are batched. They tend to have good random read performance and awesome sequential scan performance.

It comes down to using the right tool for the job. If you don't need transactions, Bolt is probably overkill. There's whole section on the BoltDB README about when not to use Bolt:

https://github.com/boltdb/bolt#caveats--limitations

> It is just badly implemented, acquires a global mutex lock across all reads and writes.

You're welcome to your opinion about it being "badly implemented" but the global mutex lock across reads and writes is simply untrue. Writes are serialized so those have a database-wide lock. However, read transactions only briefly take a lock when they start and again when they stop so they can obtain a snapshot of the root node. That gives the transaction a point-in-time snapshot of the entire database for the length of the transaction without blocking other read or write transactions.