Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TerarkDB, ByteDance's RocksDB replacement

github.com

Post reply on HN
Print view View original

61–70 of 84 posts

Re: TerarkDB, ByteDance's RocksDB replacement

#61

Post by ksec »

TerarkDB is a RocksDB replacement at ByteDance, optimized for tail latency, throughput and compression.

RocksDB is a fork of Google's LevelDB [1] at Facebook. Optimized to exploit many CPU cores, and make efficient use of fast storage, such as solid-state drives (SSD), for input/output (I/O) bound workloads.

LevelDB [2] is an open-source on-disk key-value store written by Google fellows Jeffrey Dean and Sanjay Ghemawat. Inspired by Bigtable.

Bigtable [3] is a compressed, high performance, proprietary data storage system built on Google File System, Chubby Lock Service, SSTable (log-structured storage like LevelDB) and a few other Google technologies.

I hope the info above is useful for others as I dont have the slightest idea what RockDB is, but I do know LevelDB.

[1] https://en.wikipedia.org/wiki/RocksDB

[2] https://en.wikipedia.org/wiki/LevelDB

[3] https://en.wikipedia.org/wiki/Bigtable

Re: TerarkDB, ByteDance's RocksDB replacement

#62

Post by ivzhh »

sudeepj wrote: post #8
royguo1988 wrote: post #2

TerarkDB was acquired by Bytedance two years ago and is now using widely in Bytedance's database services. I am one of the maintainers of this project you can ask any question here.

Why not merge the improvements into RocksDB itself?

Leadership or steering committee is a key factor for open source projects operated by companies. A closed pull request with comment "We won't accept the pull request because ..." should not be on the trajectory of an infrastructure project, which is to be/being widely used by any giant vendor.

So RocksDB came from LevelDB and here we go again.

Re: TerarkDB, ByteDance's RocksDB replacement

#63

Post by joshspankit »

ksec wrote: post #61

TerarkDB is a RocksDB replacement at ByteDance, optimized for tail latency, throughput and compression. RocksDB is a fork of Google's LevelDB [1] at Facebook. Optimized to exploit many CPU cores, and make efficient use of fast storage, such as solid-state drives (SSD), for input/output (I/O) bound workloads. LevelDB [2] is an open-source on-disk key-value store written by Google fellows Jeffrey Dean and Sanjay Ghemaw…

Excellent, thank you. I wish more announcements had this type of breakdown.

Re: TerarkDB, ByteDance's RocksDB replacement

#64

Post by haar »

setr wrote: post #22
ssakamoto wrote: post #18

Earlier quoted context omitted.

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

Sure it is; it’s exactly equivalent to something like forking Linux with the reasoning “I want to be the BDFL now” — eg the nvim fork

Wasn't the driver for nvim specifically disagreements with the direction/priorities/steer of the project? Is progress in a different direction necessarily a bad thing, especially if that effort couldn't be directly applied to the original anyway?

Please someone feel free to correct me, but if I recall correctly a lot of the improvements in Vim 8 were a result of the popularity of functionality in NeoVim?

Re: TerarkDB, ByteDance's RocksDB replacement

#65

Post by dboreham »

ksec wrote: post #61

TerarkDB is a RocksDB replacement at ByteDance, optimized for tail latency, throughput and compression. RocksDB is a fork of Google's LevelDB [1] at Facebook. Optimized to exploit many CPU cores, and make efficient use of fast storage, such as solid-state drives (SSD), for input/output (I/O) bound workloads. LevelDB [2] is an open-source on-disk key-value store written by Google fellows Jeffrey Dean and Sanjay Ghemaw…

Forgot the reference to BerkeleyDB ;)

Re: TerarkDB, ByteDance's RocksDB replacement

#66

Post by tgtweak »

ssakamoto wrote: post #18
royguo1988 wrote: post #9

Earlier quoted context omitted.

There are mainly three reasons here: 1. We changed the source code too much that we are not able to merge it back to RocksDB easily (This project started at 2016 as an close-source project) 2. We have different road path with RocksDB (e.g. We will remove a lot of un-used code to make TerarkDB much more light-weight than current version in the future) 3. We have lots of third-party partners (e.g. Intel, on Opatane SSD…

3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.

It feels like this is healthy, organic and very much in line with the ethos of open source to see a project take this path and arrive back in open source. If the rocks team wanted to cherry pick some compatible advancements from this project they are now free to do so.

There are much more egregious and fundamentally different violations to open source namely those you mention in your comment.

Re: TerarkDB, ByteDance's RocksDB replacement

#67

Post by wenc »

that_guy_iain wrote: post #48
reitzensteinm wrote: post #44

Earlier quoted context omitted.

Generally I agree, but if you don't know exactly what RocksDB is and why you'd want to fork it, this just isn't for you. There's probably a mutually beneficial filter being applied here by not letting beginners stumble in.

But what if RocksDB or TerakDB would solve a specific problem I'm facing and I just don't know of this solution? Lots of us have problems but don't know the exact tech stack to solve our problems; this is true at nearly every tech company I've worked for. My favourite anecdote for this was a guy basically reinvented map/reduce in the form of hacky scripts from Hadoop round about the time Hadoop and map/reduce was sta…

There are two kinds of mindsets when it comes to learning: a consumer mindset and an autodidact mindset.

In an organization, it's easy to recognize consumers -- they typically say things like: "I don't understand this. Is there a training course for this that I can sign up for?" and expect to be assigned to an internal training session or to some external course.

An autodidact on the other hand goes: "I don't understand this. Let me do some research on my own and try to teach myself."

I've been both at various junctures in my life but I've learned that in order to progress to higher levels, it's better to be an autodidact instead of a consumer. When it comes to new knowledge, there's rarely someone who will feed it to me -- I have to take the initiative to learn it myself.

There's nothing wrong in asking for a clarifying blurb (good marketing aims to make things frictionless for potential customers). But RocksDB is its own universe and it's actually pretty well known. I don't work in this space, and even I know what RocksDB is because it has come up a lot in technical discussions about storage engines. When I first encountered it, I had no idea what it was, but I gathered from comments that people were excited about it, so I googled "wiki rocksdb". It took 2 seconds.

Truly curious people are autodidacts, not consumers.

p.s. the HN comment section is a great venue to "overhear" what the community is talking about and what they find exciting. It provides a good signal to dive into certain topics. Knowledge acquisition is very much a sociological exercise as much as it is an individual one.

Re: TerarkDB, ByteDance's RocksDB replacement

#68

Post by [deleted] »

wenc wrote: post #67
that_guy_iain wrote: post #48

Earlier quoted context omitted.

But what if RocksDB or TerakDB would solve a specific problem I'm facing and I just don't know of this solution? Lots of us have problems but don't know the exact tech stack to solve our problems; this is true at nearly every tech company I've worked for. My favourite anecdote for this was a guy basically reinvented map/reduce in the form of hacky scripts from Hadoop round about the time Hadoop and map/reduce was sta…

There are two kinds of mindsets when it comes to learning: a consumer mindset and an autodidact mindset. In an organization, it's easy to recognize consumers -- they typically say things like: "I don't understand this. Is there a training course for this that I can sign up for?" and expect to be assigned to an internal training session or to some external course. An autodidact on the other hand goes: "I don't underst…

[deleted]

Re: TerarkDB, ByteDance's RocksDB replacement

#69

Post by ddorian43 »

KingOfCoders wrote: post #59
reitzensteinm wrote: post #53

Earlier quoted context omitted.

I'm not saying don't learn RocksDB - quite the opposite. It's a great tool to have in your toolbelt. I'm saying that unless you've used and hit the limits of RocksDB - and it's already absurdly fast - there's zero reason to utilize this project. Maybe it'll mature one day, have multiplatform support and a wide array of client libraries, and be to RocksDB what RocksDB was to LevelDB. But today is not that day. For now…

And how should I know this from the README? How should I decide between a project that is interesting and not explained from a project that is not intended for me?

> How should I decide between a project that is interesting and not explained from a project that is not intended for me?

You first try rocksdb/lmdb, learn it all, break things, hit limitations, and lower your standards enough to search for other things, that don't have comprehensive documentation but just a small readme/paper and checking the code.

Don't expect to build a better Postgresql on the first try.

Re: TerarkDB, ByteDance's RocksDB replacement

#70

Post by loeg »

royguo1988 wrote: post #9
sudeepj wrote: post #8

Earlier quoted context omitted.

Why not merge the improvements into RocksDB itself?

There are mainly three reasons here: 1. We changed the source code too much that we are not able to merge it back to RocksDB easily (This project started at 2016 as an close-source project) 2. We have different road path with RocksDB (e.g. We will remove a lot of un-used code to make TerarkDB much more light-weight than current version in the future) 3. We have lots of third-party partners (e.g. Intel, on Opatane SSD…

It's open source now, right? Outside of 2 and 3, could someone incorporate (some) of the improvements from TerarkDB into RocksDB? Or does it truly require some major rewrite to achieve the tail-latency benefits?

The comparison figures presented looked really impressive, thanks for sharing it.

Post reply on HN