HyperLevelDB: A High-Performance LevelDB Fork
hyperdex.org
HyperLevelDB: A High-Performance LevelDB Fork
1–10 of 30 posts
Re: HyperLevelDB: A High-Performance LevelDB Fork
#2The performance graphs look promising, but the second graph in the Parallelism section seems suspect to me. Why didn't they plot the graph out to 8 threads, like they did with the first graph?
Edit: I don't mean to sound snarky, I'm definitely impressed.
Re: HyperLevelDB: A High-Performance LevelDB Fork
#3Nice to see. I played with LevelDB a while ago, and performance seemed great as long as you didn't want any kind of durability . . . but of course you would in real life, and as soon as you start forcing things to disk LevelDB's performance dropped to levels that were beatable by any number of alternatives with fewer dependencies. To say I was unimpressed would be an understatement. It's good to see someone bringing it to where it should have been already.
Re: HyperLevelDB: A High-Performance LevelDB Fork
#4I'm glad someone is addressing these issues. I think I'm not the only person that have had some bulk-write and compaction related performance problems with LevelDB. I wonder if there is any chance the changes from this fork will be rolled back into LevelDB codebase.
Re: HyperLevelDB: A High-Performance LevelDB Fork
#5The performance graphs look promising, but the second graph in the Parallelism section seems suspect to me. Why didn't they plot the graph out to 8 threads, like they did with the first graph? Edit: I don't mean to sound snarky, I'm definitely impressed.
So we scale up to (num_cores - 1). We have 8-way and a 4-way machines. I'll update the page to make this clear.
Re: HyperLevelDB: A High-Performance LevelDB Fork
#6I'm glad someone is addressing these issues. I think I'm not the only person that have had some bulk-write and compaction related performance problems with LevelDB. I wonder if there is any chance the changes from this fork will be rolled back into LevelDB codebase.
We're doing our best to keep API compatibility with LevelDB and will do our best to keep true to LevelDB. That being said, it'd be great to see our changes flow upstream.
Re: HyperLevelDB: A High-Performance LevelDB Fork
#7I'm glad someone is addressing these issues. I think I'm not the only person that have had some bulk-write and compaction related performance problems with LevelDB. I wonder if there is any chance the changes from this fork will be rolled back into LevelDB codebase.
I can answer the opposite question: we intend to keep in sync with upstream. Since HyperLevelDB is a drop-in replacement for LevelDB, it should be fairly easy for LevelDB users to pick it up.
Re: HyperLevelDB: A High-Performance LevelDB Fork
#8How does this compare with some of Basho's LevelDB performance work? https://github.com/basho/leveldb
Re: HyperLevelDB: A High-Performance LevelDB Fork
#9[deleted]
Re: HyperLevelDB: A High-Performance LevelDB Fork
#10There seems to be three things they are doing to reach their higher performance:
1. Multi-threaded synchronized write ordering. I'm interested in the internal synchronization mechanism...
"LevelDB uses very coarse-grained synchronization which forces all writes to proceed in an ordered, first-come-first-served fashion, effectively reducing throughput to that of a single thread. HyperLevelDB increases concurrency by allowing multiple threads to agree on the order of their respective writes, and then independently apply the writes in a manner consistent with the agreed-upon order."
2. Tuned write delay on compaction. What external instrumentation/markers are they passing into hyperleveldb to tune write delay?
"HyperLevelDB removes this artificial delay, allowing the application (in our case, HyperDex) to independently decide to delay writes, using information available outside the scope of LevelDB."
3. Tuned intra-level re-writes.
"LevelDB's compaction algorithm is not efficient, and in the "fillrand" benchmark will, on average, rewrite 3MB of data in the upper level for every 1MB of data in the lower level. HyperLevelDB avoids this waste by selecting the compaction with the smallest overhead."