Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building CockroachDB on top of RocksDB

cockroachlabs.com

Post reply on HN
Print view View original

51–60 of 79 posts

Re: Building CockroachDB on top of RocksDB

#51

Post by zzzcpan »

peterwwillis wrote: post #43

RocksDB is a fork of LevelDB, which was [in]famous for its ease of corrupting data. Did Facebook ever do anything to ensure data wouldn't corrupt, or is that still a common thing operationally? (You find it more at larger scales) Here's an example of how data corruption can suck, with (example) Riak and LevelDB. The leveldb data would corrupt often, which would leave you in a predicament. Say you had 10 nodes with a…

As far as I'm aware, none of the embedded DBs can help with fast recovery after corruption or any partial recovery. Sucks, but tolerable on local networks and small nodes. For spinning disks or far away non local nodes this of course doesn't work well and you have to implement your own data store.

Re: Building CockroachDB on top of RocksDB

#52

Post by gigatexal »

ryanworl wrote: post #20
gigatexal wrote: post #17

Earlier quoted context omitted.

Interleaved tables are best for 1:1 relationships.

Hard disagree. This is the only way in a distributed, sorted KV store to get any semblance of data locality. If Cockroach and Spanner didn't do this, they would constantly be doing 2PC for modifying data that is related but stored on different groups of machines.

We are evaluating it for production and the senior engineer we talked to from the company told us that. I’d think he knows what he’s talking about.

Re: Building CockroachDB on top of RocksDB

#53

Post by gigatexal »

gigatexal wrote: post #52
ryanworl wrote: post #20

Earlier quoted context omitted.

Hard disagree. This is the only way in a distributed, sorted KV store to get any semblance of data locality. If Cockroach and Spanner didn't do this, they would constantly be doing 2PC for modifying data that is related but stored on different groups of machines.

We are evaluating it for production and the senior engineer we talked to from the company told us that. I’d think he knows what he’s talking about.

Also they do 2PC work in a 1PC way by maintaining a hidden transaction status table.

Re: Building CockroachDB on top of RocksDB

#54

Post by jandrewrogers »

m0zg wrote: post #39
stingraycharles wrote: post #38

Earlier quoted context omitted.

Rocksdb is pretty good and we relied heavily on it at QuasarDB as well. Having said that, we are nowadays deploying more and more production setups with Levyx’ Helium, which scales better and directly integrates with the hardware.

Given that Helium appears to be proprietary, what kind of perf benefit are we talking about here?

I haven't used Helium specifically, but 3-5x greater throughput would be completely believable in my experience. It is an open secret that high-performance closed source storage engines can have several times the throughput of their open source equivalents on the same hardware. High-end storage engines often have sufficient throughput to consistently saturate NVMe arrays for diverse workloads, which is not something you commonly see in open source. Consequently, it is common to see closed source storage engines for people doing high-scale sensor analytics work and similar.

The source of this performance gap is architectural. The current design of RocksDB precludes it ever being legitimately high-performance in most contexts, and most other open source storage engines use a similar design. Modern high-performance storage engines also use a common architecture implemented in minor variations, you just don't see this architecture in open source much. I realize that few software engineers have the skillset and experience required to design a top-notch storage engine, but I am still surprised by the dearth of open source examples given the large value in closing this gap.

I rarely use open source storage engines in the systems I build for this reason. The CapEx/OpEx implications of using them is far too costly at scale. Fortunately, I have the approximately free option of using my own storage engine implementations.

Re: Building CockroachDB on top of RocksDB

#55

Post by ryanworl »

gigatexal wrote: post #52
ryanworl wrote: post #20

Earlier quoted context omitted.

Hard disagree. This is the only way in a distributed, sorted KV store to get any semblance of data locality. If Cockroach and Spanner didn't do this, they would constantly be doing 2PC for modifying data that is related but stored on different groups of machines.

We are evaluating it for production and the senior engineer we talked to from the company told us that. I’d think he knows what he’s talking about.

I may be misinterpreting what you mean by a 1-1 relationship, but the documentation for Cloud Spanner, Cockroach, and the old FoundationDB SQL layer all use a similar schema to the one I described in their examples. Additionally, the F1 paper describes using it in the same way in Figure 2.

Re: Building CockroachDB on top of RocksDB

#56

Post by manigandham »

dominotw wrote: post #48
manigandham wrote: post #46

Earlier quoted context omitted.

None of those are newsql other than MemSQL, which is an OLAP system that uses a custom rowstore format and parquet for columnstores. In addition to CockroachDB there's also TiDB which runs on top of TiKV which uses RocksDB.

> None of those are newsql other than MemSQL Why aren't citus, nuodb 'newsql'? > there's also TiDB One more example doesn't qualify the statement "most are built on rocksdb". I wasn't saying there is only one newsql db built on rocksdb. Of the 14 examples listed here https://en.wikipedia.org/wiki/NewSQL only 2 that you mentioned seem to be built on rocksdb.

I'm not disagreeing, RocksDB is not used by most. The statement in the blog post is not true.

Re: Building CockroachDB on top of RocksDB

#57

Post by SamReidHughes »

zzzcpan wrote: post #51
peterwwillis wrote: post #43

RocksDB is a fork of LevelDB, which was [in]famous for its ease of corrupting data. Did Facebook ever do anything to ensure data wouldn't corrupt, or is that still a common thing operationally? (You find it more at larger scales) Here's an example of how data corruption can suck, with (example) Riak and LevelDB. The leveldb data would corrupt often, which would leave you in a predicament. Say you had 10 nodes with a…

As far as I'm aware, none of the embedded DBs can help with fast recovery after corruption or any partial recovery. Sucks, but tolerable on local networks and small nodes. For spinning disks or far away non local nodes this of course doesn't work well and you have to implement your own data store.

I think the problem the parent was referring to was the database corrupting itself, or returning corrupt results, not it getting corrupted by an outside agency.

There are certain ways LSM trees can be screwed up in implementation, but the attack surface for corruption is relatively small, so I would not be surprised if that got plugged up. There's room for noobs to put in bugs, but a small enough surface area for a comprehensively cleanup to happen later. So my attitude about RocksDB is that I'm not too worried about LevelDB's history.

But I'm just a RocksDB user and have worked on LSM stores in the past, and I'm giving you my feelings and impressions, I don't have inside knowledge here.

Re: Building CockroachDB on top of RocksDB

#58

Post by ryanworl »

jandrewrogers wrote: post #54
m0zg wrote: post #39

Earlier quoted context omitted.

Given that Helium appears to be proprietary, what kind of perf benefit are we talking about here?

I haven't used Helium specifically, but 3-5x greater throughput would be completely believable in my experience. It is an open secret that high-performance closed source storage engines can have several times the throughput of their open source equivalents on the same hardware. High-end storage engines often have sufficient throughput to consistently saturate NVMe arrays for diverse workloads, which is not something…

This technique is just more IO parallelism at the physical layer due to higher concurrency while submitting IO, correct? Since NVMe and new SSDs don't hit peak throughput until very high queue depths this doesn't surprise me.

Re: Building CockroachDB on top of RocksDB

#59

Post by gigatexal »

ryanworl wrote: post #55
gigatexal wrote: post #52

Earlier quoted context omitted.

We are evaluating it for production and the senior engineer we talked to from the company told us that. I’d think he knows what he’s talking about.

I may be misinterpreting what you mean by a 1-1 relationship, but the documentation for Cloud Spanner, Cockroach, and the old FoundationDB SQL layer all use a similar schema to the one I described in their examples. Additionally, the F1 paper describes using it in the same way in Figure 2.

He said tables with a parent child relationship that is one to one meaning if I understand that right one parent ID mapping to one child ID. Its entirely possible either I’m misinterpreting him or he’s wrong but that’s what he said.

Re: Building CockroachDB on top of RocksDB

#60

Post by [deleted] »

jandrewrogers wrote: post #30
polskibus wrote: post #2

I noticed that RocksDB is used very often in OLTP scenarios. What's the OLAP equivalent of RocksDB in OLTP world? Apache Parquet? Apache Arrow? What would you use these days to create a high performance OLAP/OLHybridP engine ?

There is a practical engineering reason why the OLAP equivalent doesn't seem to exist. General purpose storage engines, and this applies to RocksDB, are like the C++ STL in that they provide good average performance across a wide range of common cases but are nowhere close to optimal if you have a well-defined type of data model and workload as your use case. You can always gain an integer factor increase in throughp…

[deleted]
Post reply on HN