Live data from Hacker News

TerarkDB, ByteDance's RocksDB replacement

github.com

51–60 of 84 posts

Re: TerarkDB, ByteDance's RocksDB replacement

#51
post #50

TerarkDB was acquired by Bytedance two years ago and is now using widely in Bytedance's database services. I am one of the maintainers of this project you can ask any question here.

Is there a backdoor for Chinese?

Given that it is open source, you can answer this question yourself?

Re: TerarkDB, ByteDance's RocksDB replacement

#52

Earlier quoted context omitted.

There are mainly three reasons here: 1. We changed the source code too much that we are not able to merge it back to RocksDB easily (This project started at 2016 as an close-source project) 2. We have different road path with RocksDB (e.g. We will remove a lot of un-used code to make TerarkDB much more light-weight than current version in the future) 3. We have lots of third-party partners (e.g. Intel, on Opatane SSD…

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

(3) is exactly how SQLite is developed

Re: TerarkDB, ByteDance's RocksDB replacement

#53

Earlier quoted context omitted.

Generally I agree, but if you don't know exactly what RocksDB is and why you'd want to fork it, this just isn't for you. There's probably a mutually beneficial filter being applied here by not letting beginners stumble in.

But what if RocksDB or TerakDB would solve a specific problem I'm facing and I just don't know of this solution? Lots of us have problems but don't know the exact tech stack to solve our problems; this is true at nearly every tech company I've worked for. My favourite anecdote for this was a guy basically reinvented map/reduce in the form of hacky scripts from Hadoop round about the time Hadoop and map/reduce was sta…

I'm not saying don't learn RocksDB - quite the opposite. It's a great tool to have in your toolbelt.

I'm saying that unless you've used and hit the limits of RocksDB - and it's already absurdly fast - there's zero reason to utilize this project.

Maybe it'll mature one day, have multiplatform support and a wide array of client libraries, and be to RocksDB what RocksDB was to LevelDB. But today is not that day.

For now, developers that don't immediately understand what this project is for would best be served with a simple link to RocksDB.

Re: TerarkDB, ByteDance's RocksDB replacement

#54

Earlier quoted context omitted.

Generally I agree, but if you don't know exactly what RocksDB is and why you'd want to fork it, this just isn't for you. There's probably a mutually beneficial filter being applied here by not letting beginners stumble in.

"but if you don't know exactly what RocksDB" So things I don't know exactly about are not for me? That seriosly hinders my personal development. As a two decade CTO I think I should be interested in things I don't know about.

You gave advice to the project on how to attract users that don't know what RocksDB is. I'm saying that they probably don't want users that don't know what RocksDB is.

Re: TerarkDB, ByteDance's RocksDB replacement

#55

Earlier quoted context omitted.

There are mainly three reasons here: 1. We changed the source code too much that we are not able to merge it back to RocksDB easily (This project started at 2016 as an close-source project) 2. We have different road path with RocksDB (e.g. We will remove a lot of un-used code to make TerarkDB much more light-weight than current version in the future) 3. We have lots of third-party partners (e.g. Intel, on Opatane SSD…

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

Amazon did not fork mongodb, they won’t touch AGPL code, they reimplemented the server side protocol and a backend implementation on top of postgresql afaics.

Re: TerarkDB, ByteDance's RocksDB replacement

#56

Earlier quoted context omitted.

You may want to update the TerarkDb entry on dbdb.io.

Thanks, tried to log-in & reset my password but didn't receive reset email.

Email me (pavlo@cs.cmu.edu). I don't think you ever had an account.

Re: TerarkDB, ByteDance's RocksDB replacement

#57
Some of the benefits of this come from separation of values from the keys. This is an increasingly widely used technique: it is described in the WiscKey paper [1] and is also used in the PingCAP fork of RocksDB. It seems Chinese companies like forking RocksDB, I am not sure why, perhaps the combination of firewall+language barrier just makes it easier to fork and move fast than try to work regularly with upstream.

By separating out large values there's less write amplification and things get faster because more of the SSTs fit in RAM cache. RocksDB wasn't historically a great choice to hold things like file uploads - you'd use the traditional filesystem for that. But that's quite constraining. When large values work better, it not only is a performance increase, but it enables new software designs too.

[1] https://www.usenix.org/system/files/conference/fast16/fast16...

Re: TerarkDB, ByteDance's RocksDB replacement

#59

Earlier quoted context omitted.

But what if RocksDB or TerakDB would solve a specific problem I'm facing and I just don't know of this solution? Lots of us have problems but don't know the exact tech stack to solve our problems; this is true at nearly every tech company I've worked for. My favourite anecdote for this was a guy basically reinvented map/reduce in the form of hacky scripts from Hadoop round about the time Hadoop and map/reduce was sta…

I'm not saying don't learn RocksDB - quite the opposite. It's a great tool to have in your toolbelt. I'm saying that unless you've used and hit the limits of RocksDB - and it's already absurdly fast - there's zero reason to utilize this project. Maybe it'll mature one day, have multiplatform support and a wide array of client libraries, and be to RocksDB what RocksDB was to LevelDB. But today is not that day. For now…

And how should I know this from the README? How should I decide between a project that is interesting and not explained from a project that is not intended for me?

Re: TerarkDB, ByteDance's RocksDB replacement

#60
post #29

Earlier quoted context omitted.

Your all-in-one docs[1] refuse to render at all in firefox? Seems like a strange restriction, and disappointing that it doesn't even let you read the document in non-webkit browsers. Feels like the IE days all over again. "An error occurred. This browser is not supported, click here to learn more." [1] https://bytedance.feishu.cn/docs/doccnZmYFqHBm06BbvYgjsHHcKc

It renders on ubuntu 18.04 Firefox but also displays "This browser not supported with https://www.feishu.cn/hc/en-us/articles/360038713913 ". Probably doesn't support linux.

I get the same on Windows 10 so it is probably the browser.
Post reply on HN