Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building CockroachDB on top of RocksDB

cockroachlabs.com

Post reply on HN
Print view View original

61–70 of 79 posts

Re: Building CockroachDB on top of RocksDB

#61

Post by FridgeSeal »

georgewfraser wrote: post #28
arjunnarayan wrote: post #11

Earlier quoted context omitted.

Unfortunately most of the systems that build what you're describing are closed source (e.g. Snowflake, Microsoft SQL Server, Vertica, Teradata). There isn't an open-source project that does all of those things.

What about Presto?

Presto is more of a distributed SQL solution: you run it on a cluster of nodes, point them at your storage later, it’s more optimised at querying very large datasets and it’s not built or tuned for high performance (in terms of latency or execution time).

Re: Building CockroachDB on top of RocksDB

#62

Post by peterwwillis »

StreamBright wrote: post #49
peterwwillis wrote: post #43

RocksDB is a fork of LevelDB, which was [in]famous for its ease of corrupting data. Did Facebook ever do anything to ensure data wouldn't corrupt, or is that still a common thing operationally? (You find it more at larger scales) Here's an example of how data corruption can suck, with (example) Riak and LevelDB. The leveldb data would corrupt often, which would leave you in a predicament. Say you had 10 nodes with a…

I never experienced this with several in production Riak clusters running for years. Can you explain how to reproduce or give a link to any public forum where this was discussed?

Sure. Build about 10 classes of clusters of varying sizes, each with a dataset ranging from 100GB to a petabyte or more. Run them on shitty oversubscribed openstack clusters with a combination of ephemeral, Ceph, and SAN disks. Do replication to similar-ish clusters in different regions. Handle data for about 100 different applications that process so much data at such low latency that cloud-based databases aren't even an option. Keep adding nodes and storage to existing clusters over time.

It turns out that really unstable hardware/networks like to expose bugs. It also wasn't discussed in public forums. We paid for support and even employed Riak developers, and still we hobbled on putting out fires. I'll bet other DBs go through the same crap and keep it quiet.

Also, read the Riak documentation and you'll find the corruption recovery documentation among other hints at common failures and limitations.

Re: Building CockroachDB on top of RocksDB

#63

Post by atombender »

polskibus wrote: post #34
tristor wrote: post #33

Earlier quoted context omitted.

There's also Clickhouse [1] which seems to scale much better than Druid, and has similar architectural decisions to make it somewhat general as a columnar store for OLAP uses. Cloudflare wrote an article in the past where they compared Clickhouse and Druid and they chose Clickhouse because they could get similar performance on the same workload with 9 nodes in Clickhouse which would require hundreds for Druid. They b…

Does ClickHouse support fine grained data security (for example role A gives access only to tuples with column X==123)?

No [1]. ClickHouse is a fairly low-level tool. If you need that kind of thing, you build an ACL-aware app on top of it.

[1] https://clickhouse.yandex/docs/en/operations/access_rights/

Re: Building CockroachDB on top of RocksDB

#64

Post by wrigby »

gigatexal wrote: post #59
ryanworl wrote: post #55

Earlier quoted context omitted.

I may be misinterpreting what you mean by a 1-1 relationship, but the documentation for Cloud Spanner, Cockroach, and the old FoundationDB SQL layer all use a similar schema to the one I described in their examples. Additionally, the F1 paper describes using it in the same way in Figure 2.

He said tables with a parent child relationship that is one to one meaning if I understand that right one parent ID mapping to one child ID. Its entirely possible either I’m misinterpreting him or he’s wrong but that’s what he said.

I think the data locality benefits would hold for one to many relationships too (that is, one parent with many children).

Re: Building CockroachDB on top of RocksDB

#65

Post by jandrewrogers »

ryanworl wrote: post #58
jandrewrogers wrote: post #54

Earlier quoted context omitted.

I haven't used Helium specifically, but 3-5x greater throughput would be completely believable in my experience. It is an open secret that high-performance closed source storage engines can have several times the throughput of their open source equivalents on the same hardware. High-end storage engines often have sufficient throughput to consistently saturate NVMe arrays for diverse workloads, which is not something…

This technique is just more IO parallelism at the physical layer due to higher concurrency while submitting IO, correct? Since NVMe and new SSDs don't hit peak throughput until very high queue depths this doesn't surprise me.

I/O parallelism is necessary but far from sufficient. My own designs arbitrarily allow 64 reads and 64 writes to be in-flight concurrently per core. There is no science behind that limit beyond the fact it has worked brilliantly for many years across every type of storage. But I/O parallelism won't fix terrible scheduling.

A fast storage engine needs to eliminate most of the elements that will stall an execution pipeline. This means doing things like almost completely eliminating shared data structures and context switching. It also means designing your own execution and I/O scheduler to greatly reduce the various forms of stalling on memory ubiquitous in many designs. It is difficult to overstate the extent to which thoughtful schedule design can greatly improve throughput.

A state-of-the-art storage engine can drive 2+ GB/s per core, and schedule things to keep the storage hardware performance close to theoretical while smoothing out transients. It is very easy to run out of storage bandwidth in my experience.

Re: Building CockroachDB on top of RocksDB

#66

Post by viraptor »

peterwwillis wrote: post #62
StreamBright wrote: post #49

Earlier quoted context omitted.

I never experienced this with several in production Riak clusters running for years. Can you explain how to reproduce or give a link to any public forum where this was discussed?

Sure. Build about 10 classes of clusters of varying sizes, each with a dataset ranging from 100GB to a petabyte or more. Run them on shitty oversubscribed openstack clusters with a combination of ephemeral, Ceph, and SAN disks. Do replication to similar-ish clusters in different regions. Handle data for about 100 different applications that process so much data at such low latency that cloud-based databases aren't ev…

> It turns out that really unstable hardware/networks like to expose bugs.

This sounds like a weird complaint to be honest. If you verify that your hardware is unstable, how can you expect the software not to fail and corrupt data?

> Also, read the Riak documentation and you'll find the corruption recovery documentation among other hints at common failures and limitations.

I'm not sure how that's a negative thing. You think about and document recovery even if you don't expect things to fail.

Re: Building CockroachDB on top of RocksDB

#67

Post by manigandham »

peterwwillis wrote: post #62
StreamBright wrote: post #49

Earlier quoted context omitted.

I never experienced this with several in production Riak clusters running for years. Can you explain how to reproduce or give a link to any public forum where this was discussed?

Sure. Build about 10 classes of clusters of varying sizes, each with a dataset ranging from 100GB to a petabyte or more. Run them on shitty oversubscribed openstack clusters with a combination of ephemeral, Ceph, and SAN disks. Do replication to similar-ish clusters in different regions. Handle data for about 100 different applications that process so much data at such low latency that cloud-based databases aren't ev…

What? It seems like you're blaming the db software for a range of external hardware and operations issues. The apps need "so much data at such low latency" yet couldn't have a proper running environment?

Re: Building CockroachDB on top of RocksDB

#68

Post by infinite8s »

jandrewrogers wrote: post #54
m0zg wrote: post #39

Earlier quoted context omitted.

Given that Helium appears to be proprietary, what kind of perf benefit are we talking about here?

I haven't used Helium specifically, but 3-5x greater throughput would be completely believable in my experience. It is an open secret that high-performance closed source storage engines can have several times the throughput of their open source equivalents on the same hardware. High-end storage engines often have sufficient throughput to consistently saturate NVMe arrays for diverse workloads, which is not something…

What are your thoughts on LMDB?

Re: Building CockroachDB on top of RocksDB

#69

Post by ccleve »

jandrewrogers wrote: post #65
ryanworl wrote: post #58

Earlier quoted context omitted.

This technique is just more IO parallelism at the physical layer due to higher concurrency while submitting IO, correct? Since NVMe and new SSDs don't hit peak throughput until very high queue depths this doesn't surprise me.

I/O parallelism is necessary but far from sufficient. My own designs arbitrarily allow 64 reads and 64 writes to be in-flight concurrently per core. There is no science behind that limit beyond the fact it has worked brilliantly for many years across every type of storage. But I/O parallelism won't fix terrible scheduling. A fast storage engine needs to eliminate most of the elements that will stall an execution pipe…

I'd love to learn more. I'm in need of a fast engine.

What proprietary engines do you know of that I can look at?

Do you have more details on your own designs? Anything you can share?

Re: Building CockroachDB on top of RocksDB

#70

Post by nicolast »

viraptor wrote: post #66
peterwwillis wrote: post #62

Earlier quoted context omitted.

Sure. Build about 10 classes of clusters of varying sizes, each with a dataset ranging from 100GB to a petabyte or more. Run them on shitty oversubscribed openstack clusters with a combination of ephemeral, Ceph, and SAN disks. Do replication to similar-ish clusters in different regions. Handle data for about 100 different applications that process so much data at such low latency that cloud-based databases aren't ev…

> It turns out that really unstable hardware/networks like to expose bugs. This sounds like a weird complaint to be honest. If you verify that your hardware is unstable, how can you expect the software not to fail and corrupt data? > Also, read the Riak documentation and you'll find the corruption recovery documentation among other hints at common failures and limitations. I'm not sure how that's a negative thing. Yo…

> If you verify that your hardware is unstable, how can you expect the software not to fail and corrupt data?

No such thing as 'stable hardware' at scale.

Post reply on HN