Building CockroachDB on top of RocksDB • Hacker News Board

Re: Building CockroachDB on top of RocksDB

#41

Post by stingraycharles » Fri, Jan 18, 2019, 7:59 PM UTC

m0zg wrote: post #39

stingraycharles wrote: post #38

Earlier quoted context omitted.

Rocksdb is pretty good and we relied heavily on it at QuasarDB as well. Having said that, we are nowadays deploying more and more production setups with Levyx’ Helium, which scales better and directly integrates with the hardware.

Given that Helium appears to be proprietary, what kind of perf benefit are we talking about here?

In our testing, it’s multiple times faster, especially at scale. RocksDB’s compaction becomes a bottleneck fairly quickly when put under strain for extended periods of time.

Helium performs much, much better at scale and doesn’t have compaction issues. It’s proprietary, but in my experience it’s money well spent.

For the record, we were able to fully saturate a 4xNVMe with a 96 core server using Helium, while RocksDB achieved about 20% of the full NVMe capacity.

As with all benchmarks, YMMV.

Re: Building CockroachDB on top of RocksDB

#42

Post by dominotw » Fri, Jan 18, 2019, 8:23 PM UTC

nindalf wrote: post #36

dominotw wrote: post #35

> If you surveyed most NewSQL databases today, most of them are built on top of an LSM, namely, RocksDB. Is this actually true? spark, foundationdb, memsql, nuodb , citus . I am not sure any of these are built on top of rocksdb. Which ones are actually built on lsm?

Cassandra, MongoDB, BigTable, InfluxDB, LevelDB.

> Cassandra, MongoDB, BigTable, InfluxDB, LevelDB.

None of these are NewSql[1](ACID and SQL) though.

1. https://en.wikipedia.org/wiki/NewSQL

Re: Building CockroachDB on top of RocksDB

#43

Post by peterwwillis » Fri, Jan 18, 2019, 8:45 PM UTC

RocksDB is a fork of LevelDB, which was [in]famous for its ease of corrupting data. Did Facebook ever do anything to ensure data wouldn't corrupt, or is that still a common thing operationally? (You find it more at larger scales)

Here's an example of how data corruption can suck, with (example) Riak and LevelDB. The leveldb data would corrupt often, which would leave you in a predicament. Say you had 10 nodes with a 3 node replication factor, and the whole cluster is humming away at a decent clip. Now one node's leveldb corrupts, and you have to rebuild it. If you have a huge fuckoff dataset, this can take a while. Now another node goes down. Now only 1 node has the data you need, and 2 nodes are down - so now 8 nodes are doing the work of 10, and if you have any more failures, your data might be gone. Now add replication, which will suck performance and bandwidth away from the regular work. And because it would corrupt so easily & often, there needed to be hash trees to quickly identify what data was corrupt, and then you needed to fix it and rebuild your hash trees. This would also suck away performance. Finally, you can't just add new nodes while rebuilding, because the extra load makes the cluster fall over. And the more nodes, the higher the likelihood of failures.

Re: Building CockroachDB on top of RocksDB

#44

Post by ddorian43 » Fri, Jan 18, 2019, 8:55 PM UTC

stingraycharles wrote: post #41

m0zg wrote: post #39

Earlier quoted context omitted.

Given that Helium appears to be proprietary, what kind of perf benefit are we talking about here?

In our testing, it’s multiple times faster, especially at scale. RocksDB’s compaction becomes a bottleneck fairly quickly when put under strain for extended periods of time. Helium performs much, much better at scale and doesn’t have compaction issues. It’s proprietary, but in my experience it’s money well spent. For the record, we were able to fully saturate a 4xNVMe with a 96 core server using Helium, while RocksDB…

Did you try LMDB ?

Re: Building CockroachDB on top of RocksDB

#45

Post by teacpde » Fri, Jan 18, 2019, 9:15 PM UTC

peterwwillis wrote: post #43

RocksDB is a fork of LevelDB, which was [in]famous for its ease of corrupting data. Did Facebook ever do anything to ensure data wouldn't corrupt, or is that still a common thing operationally? (You find it more at larger scales) Here's an example of how data corruption can suck, with (example) Riak and LevelDB. The leveldb data would corrupt often, which would leave you in a predicament. Say you had 10 nodes with a…

Curious what is the cause of data corruption in leveldb?

Re: Building CockroachDB on top of RocksDB

#46

Post by manigandham » Fri, Jan 18, 2019, 9:43 PM UTC

dominotw wrote: post #35

> If you surveyed most NewSQL databases today, most of them are built on top of an LSM, namely, RocksDB. Is this actually true? spark, foundationdb, memsql, nuodb , citus . I am not sure any of these are built on top of rocksdb. Which ones are actually built on lsm?

None of those are newsql other than MemSQL, which is an OLAP system that uses a custom rowstore format and parquet for columnstores.

In addition to CockroachDB there's also TiDB which runs on top of TiKV which uses RocksDB.

Re: Building CockroachDB on top of RocksDB

#47

Post by nickpsecurity » Fri, Jan 18, 2019, 9:44 PM UTC

peterwwillis wrote: post #43

RocksDB is a fork of LevelDB, which was [in]famous for its ease of corrupting data. Did Facebook ever do anything to ensure data wouldn't corrupt, or is that still a common thing operationally? (You find it more at larger scales) Here's an example of how data corruption can suck, with (example) Riak and LevelDB. The leveldb data would corrupt often, which would leave you in a predicament. Say you had 10 nodes with a…

That kind of concern is probably why FoundationDB built on and modified SQLite. Its reliability is already great.

Re: Building CockroachDB on top of RocksDB

#48

Post by dominotw » Fri, Jan 18, 2019, 9:51 PM UTC

manigandham wrote: post #46

dominotw wrote: post #35

> If you surveyed most NewSQL databases today, most of them are built on top of an LSM, namely, RocksDB. Is this actually true? spark, foundationdb, memsql, nuodb , citus . I am not sure any of these are built on top of rocksdb. Which ones are actually built on lsm?

None of those are newsql other than MemSQL, which is an OLAP system that uses a custom rowstore format and parquet for columnstores. In addition to CockroachDB there's also TiDB which runs on top of TiKV which uses RocksDB.

> None of those are newsql other than MemSQL

Why aren't citus, nuodb 'newsql'?

> there's also TiDB

One more example doesn't qualify the statement "most are built on rocksdb". I wasn't saying there is only one newsql db built on rocksdb.

Of the 14 examples listed here https://en.wikipedia.org/wiki/NewSQL

only 2 that you mentioned seem to be built on rocksdb.

Re: Building CockroachDB on top of RocksDB

#49

Post by StreamBright » Fri, Jan 18, 2019, 9:56 PM UTC

peterwwillis wrote: post #43

RocksDB is a fork of LevelDB, which was [in]famous for its ease of corrupting data. Did Facebook ever do anything to ensure data wouldn't corrupt, or is that still a common thing operationally? (You find it more at larger scales) Here's an example of how data corruption can suck, with (example) Riak and LevelDB. The leveldb data would corrupt often, which would leave you in a predicament. Say you had 10 nodes with a…

I never experienced this with several in production Riak clusters running for years. Can you explain how to reproduce or give a link to any public forum where this was discussed?

Re: Building CockroachDB on top of RocksDB

#50

Post by StreamBright » Fri, Jan 18, 2019, 9:59 PM UTC

polskibus wrote: post #2

I noticed that RocksDB is used very often in OLTP scenarios. What's the OLAP equivalent of RocksDB in OLTP world? Apache Parquet? Apache Arrow? What would you use these days to create a high performance OLAP/OLHybridP engine ?

S3 + ORC|Parquet + PrestoDB works very well