61–70 of 101 posts
Figures per minute? Same-process key-value should give millions per second operations (using one thread). And accessing via network, e.g. a la Redis, should be hundreds of thousands per second [1]. [1] https://redis.io/topics/benchmarks
This is pretty exciting. I would love to see comparisons between badger and goleveldb
Why not boltdb?
(Badger author here) BoltDB is super slow. Doesn't even come close to the performance we need, which is why we decided to write Badger from scratch.
nobody has said it so I must. Badgers? We don't need no stinking badgers!
Worth noting Badger here is a reference to the Wisconsin Badgers, since it's based on a paper from UW-Madison.
Earlier quoted context omitted.
(Badger author here) BoltDB is super slow. Doesn't even come close to the performance we need, which is why we decided to write Badger from scratch.
I'm curious if you have any more details here, or comparable benchmarks to share. BoltDB uses B+ trees and you say that a B+ tree approach is worth investigating due to improvements in SSD random write performance, so does BoltDB falsify that hypothesis or do you think it's just not well implemented and the idea still has potential?
RocksDB performs much better and is the most popular and efficient KV store in the market, being used at both Google (Leveldb) and Facebook. Therefore, the benchmarks are against that. Without spending time generating benchmarks, I'll bet that Badger would beat BoltDB any day.
The idea of using B+-trees with value log has potential. One would need to do some obvious optimizations to decrease the frequency of writes to disk. Because SSDs have to run garbage collection cycles, which can affect write performance in a long running task.
But, I think it would make a great research project. And if it comes out to be better than our current approach of LSM tree in read-write performance, I'd switch in a heartbeat; because I think B+-tree might be a simpler design.
Earlier quoted context omitted.
I'm curious if you have any more details here, or comparable benchmarks to share. BoltDB uses B+ trees and you say that a B+ tree approach is worth investigating due to improvements in SSD random write performance, so does BoltDB falsify that hypothesis or do you think it's just not well implemented and the idea still has potential?
We have tried with BoltDB. Its performance is really bad. It is just badly implemented, acquires a global mutex lock across all reads and writes. We wouldn't have written Badger if BoltDB worked for us. RocksDB performs much better and is the most popular and efficient KV store in the market, being used at both Google (Leveldb) and Facebook. Therefore, the benchmarks are against that. Without spending time generating…
Earlier quoted context omitted.
Badger is designed to store data on disk (like RocksDB or LevelDB), while Redis is an in-memory storage which can't store data sets larger than memory.
You can store Redis on disk too. Not that you gain anything from doing so as persistence can be better achieved by clustering and I've never ran into issues where memory was a limiting factor, even with millions of records in Redis. Frankly if memory was a limiting factor then you'd probably want your KV store separate from your application anyway rather than the embedded approach that Badger takes. I think the real…
If you want persistence, then I'd recommend persisting to disk. While I've not had this fun with Redis, I've written code that took out an entire Cassandra ring. Were stuff only in memory, it would have not been pretty. Just because something is distributed doesn't mean it's guaranteed to never go completely down.
(That said, if you're using Redis as an in-memory cache, this is a potentially acceptable tradeoff.)
Earlier quoted context omitted.
We have tried with BoltDB. Its performance is really bad. It is just badly implemented, acquires a global mutex lock across all reads and writes. We wouldn't have written Badger if BoltDB worked for us. RocksDB performs much better and is the most popular and efficient KV store in the market, being used at both Google (Leveldb) and Facebook. Therefore, the benchmarks are against that. Without spending time generating…
Very interesting, thanks for the answer! For the application and data structure I have in mind, I expect to need lots of range iteration over small values (<32 bytes?) and relatively infrequent writes, so Badger might not be a good fit. At a minimum sounds like I'd need to set up my own benchmarks.
So, you should really test your read/write ratio with the size of the keys and payloads before selecting one solution or another. Even on the Badger tests, you can see that it can vary a lot.
off topic: flipping through the dgraph code, I noticed their licensing switch from Apache 2 to AGPLv3, anyone involved around to comment? Adding a draconic open source license is an unwise decision for an early stage database product imo ( https://open.dgraph.io/licensing is a dead link)
1) copyleft versus non-copyleft
2) which copyleft license to choose
3) the strategy of the dgraph
Regarding 1), non-copyleft leads to higher short-term adoption, but copyleft is often the better choice long-term. Moreover, history has shown that if you switch from a non-copyleft to a copyleft license, people will feel tricked. So the "early stage" argument doesn't hold. If you want to use copyleft long-term, better be honest and do so upfront.
Regarding 2), whenever you ask a lawyer in that field, they usually tell you that AGPLv3 is almost always what you want, preferable to GPLv3 and most other alternatives. So the "draconic open source license" argument doesn't hold. AGPLv3 just closes large holes which GPLv3 left open, to ensure that people actually stick to the copyleft principle. So if you want copyleft, choose AGPLv3 unless you have a very compelling reason not to.
Regarding 3), the dgraph people seem to see it a similar way:
There is so much negativity in these comments! This project is really cool! I really appreciate the trend to rewrite C/C++ libraries in Go. It has always been really frustrating that hacky library wrappers for other languages leave performance and features on the table because they're either incomplete or just too hard to implement in the host language. For most of the languages out there, it is always better to have…
If we're going to rewrite libraries which may be used by other applications, which gets deployed on servers, where people will need to update things in the name of security...
It would be nice if those libraries weren't statically linked, and thus would need to be rebuilt, pushed and redeployed for every one of their dependent libraries which had a security issue.
The only place I can imagine Go-based software being useful, is for code you've built yourself, for use in a always rolling forward cloud-environment you maintain yourself, where updating dependencies and rebuilding is part of the day to day operations.
Trusting Go-code written by other people for anything you deploy publicly sounds like something I'd rather never do. It sounds like a security nightmare.