Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TerarkDB, ByteDance's RocksDB replacement

github.com

Post reply on HN
Print view View original

21–30 of 84 posts

Re: TerarkDB, ByteDance's RocksDB replacement

#21

Post by royguo1988 »

dikei wrote: post #17

Since TerarkDB is latency-optimized, it would be quite interesting to test it with Kafka Streaming or Flink which are currently using RocksDB for stateful stream processing. Hopefully, that will also brings better Java test coverage and integration.

We didn't test the Java Binding for quick a long time, I am not sure if it can still compile the Java Binding well, please fill an issue on Github if you find it didn't work anymore, thanks!

I remembered that we tried it on Flink in early versions are the result is pretty good.

Re: TerarkDB, ByteDance's RocksDB replacement

#22

Post by setr »

ssakamoto wrote: post #18
royguo1988 wrote: post #9

Earlier quoted context omitted.

There are mainly three reasons here: 1. We changed the source code too much that we are not able to merge it back to RocksDB easily (This project started at 2016 as an close-source project) 2. We have different road path with RocksDB (e.g. We will remove a lot of un-used code to make TerarkDB much more light-weight than current version in the future) 3. We have lots of third-party partners (e.g. Intel, on Opatane SSD…

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

Sure it is; it’s exactly equivalent to something like forking Linux with the reasoning “I want to be the BDFL now” — eg the nvim fork

Re: TerarkDB, ByteDance's RocksDB replacement

#23

Post by alexgartrell »

ssakamoto wrote: post #18
royguo1988 wrote: post #9

Earlier quoted context omitted.

There are mainly three reasons here: 1. We changed the source code too much that we are not able to merge it back to RocksDB easily (This project started at 2016 as an close-source project) 2. We have different road path with RocksDB (e.g. We will remove a lot of un-used code to make TerarkDB much more light-weight than current version in the future) 3. We have lots of third-party partners (e.g. Intel, on Opatane SSD…

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

First, it’s reeaallllyyyy expensive to invest enough in an open source project that you have a reasonable chance of steering it.

Second, even if you do the first, the whole thing gets screwed up again when you start trying to introduce vendor code into the mix. Generally, no one upstream gives a crap that you have super compelling business reasons to compromise on code quality (or even trivial things like how code is committed: tarballs vs good git hygiene), and vendors sometimes compromise a lot.

So it’s not surprising that sometimes groups choose to do the expedient thing to get something to market instead of doing things “the right way.” In a lot of respects, the original Android did this with Linux.

Competition is good.

Re: TerarkDB, ByteDance's RocksDB replacement

#24

Post by meta2meta »

royguo1988 wrote: post #2

TerarkDB was acquired by Bytedance two years ago and is now using widely in Bytedance's database services. I am one of the maintainers of this project you can ask any question here.

I see "#include " in the "memtable/terark_zip_entry_index.cc" but I can't find "cspptrie.inl" in the repo. Is the code auto-generated or not open source now?

Re: TerarkDB, ByteDance's RocksDB replacement

#25

Post by klodolph »

alexgartrell wrote: post #23
ssakamoto wrote: post #18

Earlier quoted context omitted.

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

First, it’s reeaallllyyyy expensive to invest enough in an open source project that you have a reasonable chance of steering it. Second, even if you do the first, the whole thing gets screwed up again when you start trying to introduce vendor code into the mix. Generally, no one upstream gives a crap that you have super compelling business reasons to compromise on code quality (or even trivial things like how code is…

> In a lot of respects, the original Android did this with Linux.

Android vendors keep doing this over and over again with Linux, which explains why so many phones are stuck on old versions of Android.

Re: TerarkDB, ByteDance's RocksDB replacement

#26

Post by adamretter »

dikei wrote: post #17

Since TerarkDB is latency-optimized, it would be quite interesting to test it with Kafka Streaming or Flink which are currently using RocksDB for stateful stream processing. Hopefully, that will also brings better Java test coverage and integration.

I have helped backport RocksDB Java API changes for Kafka in the past. The main issue in the integration seems to be organisational rather than technical. Kafka has a more conservative release approach, whereas RocksDB has lots of releases and the API changes frequently.

I would be happy to be contacted about specific Java API issues with RocksDB; Maybe I can help.

Re: TerarkDB, ByteDance's RocksDB replacement

#28

Post by cowsandmilk »

ssakamoto wrote: post #18
royguo1988 wrote: post #9

Earlier quoted context omitted.

There are mainly three reasons here: 1. We changed the source code too much that we are not able to merge it back to RocksDB easily (This project started at 2016 as an close-source project) 2. We have different road path with RocksDB (e.g. We will remove a lot of un-used code to make TerarkDB much more light-weight than current version in the future) 3. We have lots of third-party partners (e.g. Intel, on Opatane SSD…

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

It is perfectly in line with open source philosophy to be able to fork a project and have control over my fork. Especially given 2 where they have different goals from upstream.

Re: TerarkDB, ByteDance's RocksDB replacement

#29

Post by josephg »

royguo1988 wrote: post #14
nikhilsimha wrote: post #3

Earlier quoted context omitted.

Always love it when I see a maintainer offer clarifications in a HN comment section! What are the reasons for the perf improvements we see here?

We are working on our `all-in-one docs` right now, please watch our repo, thanks! I replied some of the reasons in previous comment.

Your all-in-one docs[1] refuse to render at all in firefox? Seems like a strange restriction, and disappointing that it doesn't even let you read the document in non-webkit browsers. Feels like the IE days all over again.

"An error occurred. This browser is not supported, click here to learn more."

[1] https://bytedance.feishu.cn/docs/doccnZmYFqHBm06BbvYgjsHHcKc

Re: TerarkDB, ByteDance's RocksDB replacement

#30

Post by tinco »

rishav_sharan wrote: post #20
ssakamoto wrote: post #18

Earlier quoted context omitted.

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

How so? Unless they are stopping normal users from committing code as well?

Even if they stopped normal users from committing it would still be adhering to open source philosophy.
Post reply on HN