Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TerarkDB, ByteDance's RocksDB replacement

github.com

Post reply on HN
Print view View original

31–40 of 84 posts

Re: TerarkDB, ByteDance's RocksDB replacement

#31

Post by random5634 »

ssakamoto wrote: post #18
royguo1988 wrote: post #9

Earlier quoted context omitted.

There are mainly three reasons here: 1. We changed the source code too much that we are not able to merge it back to RocksDB easily (This project started at 2016 as an close-source project) 2. We have different road path with RocksDB (e.g. We will remove a lot of un-used code to make TerarkDB much more light-weight than current version in the future) 3. We have lots of third-party partners (e.g. Intel, on Opatane SSD…

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

I didn't know this. How do I contribute to Oracle's Unbreakable Linux or Redhat's RHEL? I know I can fork them, but not sure how I can push my commits into their code and didn't realize that was required!

Re: TerarkDB, ByteDance's RocksDB replacement

#32

Post by ssakamoto »

random5634 wrote: post #31
ssakamoto wrote: post #18

Earlier quoted context omitted.

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

I didn't know this. How do I contribute to Oracle's Unbreakable Linux or Redhat's RHEL? I know I can fork them, but not sure how I can push my commits into their code and didn't realize that was required!

I did not say it was required. But you can always contribute.

Re: TerarkDB, ByteDance's RocksDB replacement

#33

Post by ssakamoto »

setr wrote: post #22
ssakamoto wrote: post #18

Earlier quoted context omitted.

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

Sure it is; it’s exactly equivalent to something like forking Linux with the reasoning “I want to be the BDFL now” — eg the nvim fork

No disagreements there. Contention is it's not good for the original.

Re: TerarkDB, ByteDance's RocksDB replacement

#34

Post by ssakamoto »

alexgartrell wrote: post #23
ssakamoto wrote: post #18

Earlier quoted context omitted.

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

First, it’s reeaallllyyyy expensive to invest enough in an open source project that you have a reasonable chance of steering it. Second, even if you do the first, the whole thing gets screwed up again when you start trying to introduce vendor code into the mix. Generally, no one upstream gives a crap that you have super compelling business reasons to compromise on code quality (or even trivial things like how code is…

Imagine if there were multiple incompatible and competing linux kernels. What we have now is AMD/MS/Apple etc... contributing to the kernel through "vendor code". Imagine if AMD released a AMDLinux and Nvidia had NvidiaLinux.

Re: TerarkDB, ByteDance's RocksDB replacement

#35

Post by oefrha »

josephg wrote: post #29
royguo1988 wrote: post #14

Earlier quoted context omitted.

We are working on our `all-in-one docs` right now, please watch our repo, thanks! I replied some of the reasons in previous comment.

Your all-in-one docs[1] refuse to render at all in firefox? Seems like a strange restriction, and disappointing that it doesn't even let you read the document in non-webkit browsers. Feels like the IE days all over again. "An error occurred. This browser is not supported, click here to learn more." [1] https://bytedance.feishu.cn/docs/doccnZmYFqHBm06BbvYgjsHHcKc

Works perfectly for me in Firefox (macOS). Might be a misleadingly worded "something went wrong, we don't know what" message. Probably want to check your console, could be a network problem.

Re: TerarkDB, ByteDance's RocksDB replacement

#36

Post by KingOfCoders »

What I don't understand on many pages (e.g. the TerarkDB Github README, or all the release pages linked from HN):

Why not explain your project to me first? Assume I know nothing about the project and followed a link from HN.

"TerarkDB is a RocksDB replacement"

doesn't help me, what is RocksDB?

This is a huge missed chance for projects to get new users. Start every release note with a sentence explaining your project. Assume people reading your release notes are non-users.

Re: TerarkDB, ByteDance's RocksDB replacement

#37

Post by royguo1988 »

continuations wrote: post #16
royguo1988 wrote: post #2

TerarkDB was acquired by Bytedance two years ago and is now using widely in Bytedance's database services. I am one of the maintainers of this project you can ask any question here.

I remember reading about this a few years ago here. If I remember correctly back then the main selling point was that it used succinct data structure and it was only the compression algo that was not open source - everything else was. But now when I look at the new repo and the online doc there is no mention of succinct data struct anywhere. Also, the benchmarks back then claimed 10x or more faster than RocksDB. Now…

Thanks for your attention, glad someone here still remember our history, TerarkDB is now FULLY open source with `succinct data structures`.

Here's the reasons: 1. Our `all-in-one` docs are still under writing, we will cover that part later. 2. For the performance part, we are now showing real-world cases, not a well-designed benchmark.(We selected the best result to show our work few years ago, don't want to do it anymore) 3. About why TerarkDB is faster than RocksDB will be explained in our `all-in-one docs` in one week, and most of the reasons are not magic, just engineering efforts

Thanks again for your remembering us.

Re: TerarkDB, ByteDance's RocksDB replacement

#38

Post by wanghenshui »

meta2meta wrote: post #24
royguo1988 wrote: post #2

TerarkDB was acquired by Bytedance two years ago and is now using widely in Bytedance's database services. I am one of the maintainers of this project you can ask any question here.

I see "#include " in the "memtable/terark_zip_entry_index.cc" but I can't find "cspptrie.inl" in the repo. Is the code auto-generated or not open source now?

submodule, https://github.com/bytedance/terark-zip

Re: TerarkDB, ByteDance's RocksDB replacement

#39

Post by royguo1988 »

meta2meta wrote: post #24
royguo1988 wrote: post #2

TerarkDB was acquired by Bytedance two years ago and is now using widely in Bytedance's database services. I am one of the maintainers of this project you can ask any question here.

I see "#include " in the "memtable/terark_zip_entry_index.cc" but I can't find "cspptrie.inl" in the repo. Is the code auto-generated or not open source now?

All source code is open source now. You can find them in `third-party/terark-zip`, terark-zip is a standalone repo that contains only core algorithms.

Re: TerarkDB, ByteDance's RocksDB replacement

#40

Post by royguo1988 »

supergirl wrote: post #27
royguo1988 wrote: post #2

TerarkDB was acquired by Bytedance two years ago and is now using widely in Bytedance's database services. I am one of the maintainers of this project you can ask any question here.

where is it used? what kind of data is stored in it?

In bytedance, a few database services are using TerarkDB.
Post reply on HN