Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TerarkDB, ByteDance's RocksDB replacement

github.com

Post reply on HN
Print view View original

11–20 of 84 posts

Re: TerarkDB, ByteDance's RocksDB replacement

#11

Post by royguo1988 »

yomly wrote: post #4
royguo1988 wrote: post #2

TerarkDB was acquired by Bytedance two years ago and is now using widely in Bytedance's database services. I am one of the maintainers of this project you can ask any question here.

Do you have a write up for why you got rid of RocksDB

We are working on our `all-in-one docs` which will explain everything.

I want to address that we are not meant to "get rid of" RocksDB (which lots of KV engine claimed). What we want to do is provide another solution for storage engine users with different road path (focusing on new hardware and heavy-write workloads).

For simple use cases, there will be no difference no matter what engine you use.

And for most cases, upgrade your hardware (e.g. SATA SSD to NVMe SSD) or tuning your RocksDB parameters would save you lots time, just make sure you understand what you are doing.

There's no cue for every workloads, try TerarkDB if RocksDB happens not fit your scenario.

Re: TerarkDB, ByteDance's RocksDB replacement

#13

Post by royguo1988 »

royguo1988 wrote: post #11
yomly wrote: post #4

Earlier quoted context omitted.

Do you have a write up for why you got rid of RocksDB

We are working on our `all-in-one docs` which will explain everything. I want to address that we are not meant to "get rid of" RocksDB (which lots of KV engine claimed). What we want to do is provide another solution for storage engine users with different road path (focusing on new hardware and heavy-write workloads). For simple use cases, there will be no difference no matter what engine you use. And for most cases…

The reasons we did a better job(from our own perspective) than RocksDB are: 1. We moved lots of code out side db_mutex (db mutex is convenient but costs too much) 2. We introduced a new KV separation implementation that we believe is better than RocksDB’s implementation (we didn't hear any production user are using RocksDB's KV separation yet) 3. We introduced a lazy compaction strategy that can delay compaction task while online services are dealing with short-time heavy writing. 4. Other optimizations like time histogram based TTL, pipelined WAL sync.

Re: TerarkDB, ByteDance's RocksDB replacement

#14

Post by royguo1988 »

nikhilsimha wrote: post #3
royguo1988 wrote: post #2

TerarkDB was acquired by Bytedance two years ago and is now using widely in Bytedance's database services. I am one of the maintainers of this project you can ask any question here.

Always love it when I see a maintainer offer clarifications in a HN comment section! What are the reasons for the perf improvements we see here?

We are working on our `all-in-one docs` right now, please watch our repo, thanks! I replied some of the reasons in previous comment.

Re: TerarkDB, ByteDance's RocksDB replacement

#15

Post by royguo1988 »

polskibus wrote: post #12
royguo1988 wrote: post #2

TerarkDB was acquired by Bytedance two years ago and is now using widely in Bytedance's database services. I am one of the maintainers of this project you can ask any question here.

You may want to update the TerarkDb entry on dbdb.io.

Thanks, tried to log-in & reset my password but didn't receive reset email.

Re: TerarkDB, ByteDance's RocksDB replacement

#16

Post by continuations »

royguo1988 wrote: post #2

TerarkDB was acquired by Bytedance two years ago and is now using widely in Bytedance's database services. I am one of the maintainers of this project you can ask any question here.

I remember reading about this a few years ago here. If I remember correctly back then the main selling point was that it used succinct data structure and it was only the compression algo that was not open source - everything else was.

But now when I look at the new repo and the online doc there is no mention of succinct data struct anywhere.

Also, the benchmarks back then claimed 10x or more faster than RocksDB. Now the performance claim is much more modest.

Does that mean TerarkDB no longer uses succinct data struct? Or are you just open sourcing a lower-end version of the software without the secret sauce?

Can you talk about what makes TerarkDB faster than RocksDB?

Re: TerarkDB, ByteDance's RocksDB replacement

#17

Post by dikei »

Since TerarkDB is latency-optimized, it would be quite interesting to test it with Kafka Streaming or Flink which are currently using RocksDB for stateful stream processing.

Hopefully, that will also brings better Java test coverage and integration.

Re: TerarkDB, ByteDance's RocksDB replacement

#18

Post by ssakamoto »

royguo1988 wrote: post #9
sudeepj wrote: post #8

Earlier quoted context omitted.

Why not merge the improvements into RocksDB itself?

There are mainly three reasons here: 1. We changed the source code too much that we are not able to merge it back to RocksDB easily (This project started at 2016 as an close-source project) 2. We have different road path with RocksDB (e.g. We will remove a lot of un-used code to make TerarkDB much more light-weight than current version in the future) 3. We have lots of third-party partners (e.g. Intel, on Opatane SSD…

3) is not in line with an open source philosophy.

EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

Re: TerarkDB, ByteDance's RocksDB replacement

#19

Post by ssakamoto »

royguo1988 wrote: post #13
royguo1988 wrote: post #11

Earlier quoted context omitted.

We are working on our `all-in-one docs` which will explain everything. I want to address that we are not meant to "get rid of" RocksDB (which lots of KV engine claimed). What we want to do is provide another solution for storage engine users with different road path (focusing on new hardware and heavy-write workloads). For simple use cases, there will be no difference no matter what engine you use. And for most cases…

The reasons we did a better job(from our own perspective) than RocksDB are: 1. We moved lots of code out side db_mutex (db mutex is convenient but costs too much) 2. We introduced a new KV separation implementation that we believe is better than RocksDB’s implementation (we didn't hear any production user are using RocksDB's KV separation yet) 3. We introduced a lazy compaction strategy that can delay compaction task…

Why not submit these improvements to rocksdb ?

Re: TerarkDB, ByteDance's RocksDB replacement

#20

Post by rishav_sharan »

ssakamoto wrote: post #18
royguo1988 wrote: post #9

Earlier quoted context omitted.

There are mainly three reasons here: 1. We changed the source code too much that we are not able to merge it back to RocksDB easily (This project started at 2016 as an close-source project) 2. We have different road path with RocksDB (e.g. We will remove a lot of un-used code to make TerarkDB much more light-weight than current version in the future) 3. We have lots of third-party partners (e.g. Intel, on Opatane SSD…

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

How so? Unless they are stopping normal users from committing code as well?
Post reply on HN