Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TerarkDB, ByteDance's RocksDB replacement

github.com

Post reply on HN
Print view View original

41–50 of 84 posts

Re: TerarkDB, ByteDance's RocksDB replacement

#41

Post by nextaccountic »

ssakamoto wrote: post #18
royguo1988 wrote: post #9

Earlier quoted context omitted.

There are mainly three reasons here: 1. We changed the source code too much that we are not able to merge it back to RocksDB easily (This project started at 2016 as an close-source project) 2. We have different road path with RocksDB (e.g. We will remove a lot of un-used code to make TerarkDB much more light-weight than current version in the future) 3. We have lots of third-party partners (e.g. Intel, on Opatane SSD…

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

> Eg. Amazon forking and selling MongoDB.

Are they giving back the source? And letting Mongo merge their changes if they wish?

Because that's what open source is all about.

Re: TerarkDB, ByteDance's RocksDB replacement

#42

Post by continuations »

royguo1988 wrote: post #37
continuations wrote: post #16

Earlier quoted context omitted.

I remember reading about this a few years ago here. If I remember correctly back then the main selling point was that it used succinct data structure and it was only the compression algo that was not open source - everything else was. But now when I look at the new repo and the online doc there is no mention of succinct data struct anywhere. Also, the benchmarks back then claimed 10x or more faster than RocksDB. Now…

Thanks for your attention, glad someone here still remember our history, TerarkDB is now FULLY open source with `succinct data structures`. Here's the reasons: 1. Our `all-in-one` docs are still under writing, we will cover that part later. 2. For the performance part, we are now showing real-world cases, not a well-designed benchmark.(We selected the best result to show our work few years ago, don't want to do it an…

Great. Looking forward to learn more about this.

Re: TerarkDB, ByteDance's RocksDB replacement

#43

Post by supergirl »

royguo1988 wrote: post #40
supergirl wrote: post #27

Earlier quoted context omitted.

where is it used? what kind of data is stored in it?

In bytedance, a few database services are using TerarkDB.

yes, I got that :) but can you say more? what kind of database services? what data is stored, what is the scale, what are the requirements, etc.

Re: TerarkDB, ByteDance's RocksDB replacement

#44

Post by reitzensteinm »

KingOfCoders wrote: post #36

What I don't understand on many pages (e.g. the TerarkDB Github README, or all the release pages linked from HN): Why not explain your project to me first? Assume I know nothing about the project and followed a link from HN. "TerarkDB is a RocksDB replacement" doesn't help me, what is RocksDB? This is a huge missed chance for projects to get new users. Start every release note with a sentence explaining your project.…

Generally I agree, but if you don't know exactly what RocksDB is and why you'd want to fork it, this just isn't for you.

There's probably a mutually beneficial filter being applied here by not letting beginners stumble in.

Re: TerarkDB, ByteDance's RocksDB replacement

#45

Post by KingOfCoders »

reitzensteinm wrote: post #44
KingOfCoders wrote: post #36

What I don't understand on many pages (e.g. the TerarkDB Github README, or all the release pages linked from HN): Why not explain your project to me first? Assume I know nothing about the project and followed a link from HN. "TerarkDB is a RocksDB replacement" doesn't help me, what is RocksDB? This is a huge missed chance for projects to get new users. Start every release note with a sentence explaining your project.…

Generally I agree, but if you don't know exactly what RocksDB is and why you'd want to fork it, this just isn't for you. There's probably a mutually beneficial filter being applied here by not letting beginners stumble in.

"but if you don't know exactly what RocksDB"

So things I don't know exactly about are not for me? That seriosly hinders my personal development.

As a two decade CTO I think I should be interested in things I don't know about.

Re: TerarkDB, ByteDance's RocksDB replacement

#46

Post by royguo1988 »

supergirl wrote: post #43
royguo1988 wrote: post #40

Earlier quoted context omitted.

In bytedance, a few database services are using TerarkDB.

yes, I got that :) but can you say more? what kind of database services? what data is stored, what is the scale, what are the requirements, etc.

Sorry for the unclear response. 1) We use TerarkDB under a distributed SQL database and TerarkDB helps to store its pages (16KB page), its one of the most widely used SQL database inside Bytedance. 2) We use TerarkDB under a Redis compatible distributed cache system to store raw key value pairs.

Almost all kinds of workloads are here since TerarkDB runs under too many database clusters (each cluster only serves a single application)

Re: TerarkDB, ByteDance's RocksDB replacement

#47

Post by ddorian43 »

josephg wrote: post #29
royguo1988 wrote: post #14

Earlier quoted context omitted.

We are working on our `all-in-one docs` right now, please watch our repo, thanks! I replied some of the reasons in previous comment.

Your all-in-one docs[1] refuse to render at all in firefox? Seems like a strange restriction, and disappointing that it doesn't even let you read the document in non-webkit browsers. Feels like the IE days all over again. "An error occurred. This browser is not supported, click here to learn more." [1] https://bytedance.feishu.cn/docs/doccnZmYFqHBm06BbvYgjsHHcKc

It renders on ubuntu 18.04 Firefox but also displays "This browser not supported with https://www.feishu.cn/hc/en-us/articles/360038713913". Probably doesn't support linux.

Re: TerarkDB, ByteDance's RocksDB replacement

#48

Post by that_guy_iain »

reitzensteinm wrote: post #44
KingOfCoders wrote: post #36

What I don't understand on many pages (e.g. the TerarkDB Github README, or all the release pages linked from HN): Why not explain your project to me first? Assume I know nothing about the project and followed a link from HN. "TerarkDB is a RocksDB replacement" doesn't help me, what is RocksDB? This is a huge missed chance for projects to get new users. Start every release note with a sentence explaining your project.…

Generally I agree, but if you don't know exactly what RocksDB is and why you'd want to fork it, this just isn't for you. There's probably a mutually beneficial filter being applied here by not letting beginners stumble in.

But what if RocksDB or TerakDB would solve a specific problem I'm facing and I just don't know of this solution? Lots of us have problems but don't know the exact tech stack to solve our problems; this is true at nearly every tech company I've worked for. My favourite anecdote for this was a guy basically reinvented map/reduce in the form of hacky scripts from Hadoop round about the time Hadoop and map/reduce was starting to get traction.

Re: TerarkDB, ByteDance's RocksDB replacement

#49

Post by ssakamoto »

cowsandmilk wrote: post #28
ssakamoto wrote: post #18

Earlier quoted context omitted.

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

It is perfectly in line with open source philosophy to be able to fork a project and have control over my fork. Especially given 2 where they have different goals from upstream.

No necessarily true in this case. They are compatible and it's merely a performance improvements from the code.
Post reply on HN