81–90 of 101 posts
There is so much negativity in these comments! This project is really cool! I really appreciate the trend to rewrite C/C++ libraries in Go. It has always been really frustrating that hacky library wrappers for other languages leave performance and features on the table because they're either incomplete or just too hard to implement in the host language. For most of the languages out there, it is always better to have…
Now let somebody run a QuickCheck on it, like they did with LevelDB: http://htmlpreview.github.io/?https://raw.github.com/strange...
Why not boltdb?
(Badger author here) BoltDB is super slow. Doesn't even come close to the performance we need, which is why we decided to write Badger from scratch.
If huge, long-running transactions aren't needed, then it may be easier to just let transactions happen in RAM, with a cache to enforce isolation and emulate atomicity. So for exmaple, while a transaction is committing, the system would need to cache the previous (that is, currently committed) version of every key, to avoid having other transactions reading the on-disk data, which is in the process of being updated. It would also need either a redo log or undo log so that, in the event of a cache, any half-written transactions can be either replayed or rolled back.
Earlier quoted context omitted.
I think what fortytw2 was trying to say is that the AGPL is not a wise choice for software that wants to gain the most popularity and usage as possible since usage of AGPL licensed software is categorically banned (even more so than GPLV3) by a some of companies. As an aside, one can certainly describe something as draconic(or whatever else) if one views it as such; it's just an opinion.
Having just gone through a lengthy review process identifying a suitable license for our soon to be open-source software which has a commercial aspect, and selected AGPLv3, I'm very curious to know what companies have it "categorically banned". We did some research and didn't find that anyone had an issue with it. Whilst AGPL does open up come grey-areas which aren't as well understood as GPL the general reason to us…
I would also be surprised if Apple (being so allergic to the GPLv3) used AGPLv3 software, though that's just speculation.
Earlier quoted context omitted.
redis is not an embedded database, while rocksdb/boltdb/badger are.
Can you say what exactly is an "embedded workload"? I have seen this a few times now and tried googling but only ever come up with references to embedded systems and I'm guessing that this is not the same context. I know rocksdb is LSM-based and was built by FB to address write amplification on SSDs.
Badger, RocksDB, LMDB, etc. are storage engines. A process uses these storage engines to write data to memory (note: the storage engine may support persistent, volatile, or both types of memory).
A database management system (DBMS) is a higher level concept that often has multiple processes either on a single server or distributed across multiple servers. Simply stated, each process within a DBMS uses the storage engine to read/write to/from memory (persistent or volatile).
It's important to note that storage engines are a specialized area and require different skills from writing a DBMS. It's a big deal when other people write high quality storage engines because it makes it a lot easier to write a DBMS.
Range iteration latency is very important and might be limited by concurrency. I think you can only get 100K IOPS on Amazon’s i3.large when the disk Request queue is full. fio [1] can easily do this because it spawn a number of threads While working with Rocksdb we also found that Range iteration latency was very bad compared to a B+-tree and that RocksDB get good read performance mostly from random read because it's…
(Badger author) We have tried huge prefetch size, using one Goroutine for each key; hence 100K concurrent goroutines doing value prefetching. But, in practice, throughput stabilizes after a very small number of goroutines (like 10). I suspect it's the SSD read latency that's causing range iteration to be slow; unless we're dealing with some slowness inherent to Go. A good way to test it out would be to write fio in G…
sudo apt-get install libaio1 libaio-dev.
Probably a naive question, but how does it compare to Redis? When would someone look for a K-V store written in X instead of the already mature Redis?
An application developer would not choose Badger, but instead would pick a DBMS such as Redis.
A database engineer would use Badger to develop a DBMS that the application engineer could use. If the database engineer so chooses they could expose a Redis compatible API.
Probably a naive question, but how does it compare to Redis? When would someone look for a K-V store written in X instead of the already mature Redis?
Yes this is more comparable to level, rocks or at a higher level Cassandra, ScyllaDB, DynamoDB etc.
Cassandra, for example, has it's own storage engine that's responsible for writing bits to disk.
And possibly Go has the right approach! Is it better to make C integration as simple and smooth as possible, to leverage existing libraries, or is it better to encourage people to ditch all that unsafe C code and write everything in Go?
off topic: flipping through the dgraph code, I noticed their licensing switch from Apache 2 to AGPLv3, anyone involved around to comment? Adding a draconic open source license is an unwise decision for an early stage database product imo ( https://open.dgraph.io/licensing is a dead link)