Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building CockroachDB on top of RocksDB

cockroachlabs.com

Post reply on HN
Print view View original

11–20 of 79 posts

Re: Building CockroachDB on top of RocksDB

#11

Post by arjunnarayan »

ryanworl wrote: post #10
polskibus wrote: post #6

Earlier quoted context omitted.

I know there are many techniques that used together give good performance (optimal memory layout, compression, vectorization, etc. etc.), however I'd like to use a package that does a lot of it, same what RocksDB (or SQLite) does for OLTP cases. Is there something like that? If not, what's out there that gives the best foundation for building OLAP functionalities on top of it?

No, I am not aware of any storage engine that provides that out of the box. The techniques are very tied into what your query processing engine can do and expects the data to look like. For example, do you materialize tuples immediately, or do you fully run it through your processing pipeline and not materialize until the end? Your storage engine and format needs to be at least somewhat involved in answer that questi…

Unfortunately most of the systems that build what you're describing are closed source (e.g. Snowflake, Microsoft SQL Server, Vertica, Teradata). There isn't an open-source project that does all of those things.

Re: Building CockroachDB on top of RocksDB

#12

Post by dominotw »

polskibus wrote: post #2

I noticed that RocksDB is used very often in OLTP scenarios. What's the OLAP equivalent of RocksDB in OLTP world? Apache Parquet? Apache Arrow? What would you use these days to create a high performance OLAP/OLHybridP engine ?

> I noticed that RocksDB is used very often in OLTP scenarios.

My experience with it has been most stream processing in kafka streams ect as local state store.

Re: Building CockroachDB on top of RocksDB

#14

Post by arjunnarayan »

perfmode wrote: post #13

When a SQL implementation is built on a KV storage engine, how do tables, rows, and columns typically map to the underlying KV data model?

Excellent question. There's a CockroachDB blog post about that: https://www.cockroachlabs.com/blog/sql-in-cockroachdb-mappin...

Re: Building CockroachDB on top of RocksDB

#16

Post by ryanworl »

perfmode wrote: post #13

When a SQL implementation is built on a KV storage engine, how do tables, rows, and columns typically map to the underlying KV data model?

The CockroachDB blog post on this topic is a good summary. There is an additional trick that isn't directly KV related, but is important in a distributed environment when using a KV storage engine.

When defining a hierarchy of tables, such as customers -> orders -> order_line_items, you can make the primary key of the child tables contain the primary key of the parent table.

e.g. (customer_id) for the customers table, (customer_id, order_id) for the orders table, then (customer_id, order_id, line_item_id) for the order_line_items.

When this is stored on disk in a sorted format, it makes joins between these extremely cheap because the data will all be next to each other on disk.

CockroachDB calls this "interleaved tables".

Re: Building CockroachDB on top of RocksDB

#17

Post by gigatexal »

ryanworl wrote: post #16
perfmode wrote: post #13

When a SQL implementation is built on a KV storage engine, how do tables, rows, and columns typically map to the underlying KV data model?

The CockroachDB blog post on this topic is a good summary. There is an additional trick that isn't directly KV related, but is important in a distributed environment when using a KV storage engine. When defining a hierarchy of tables, such as customers -> orders -> order_line_items, you can make the primary key of the child tables contain the primary key of the parent table. e.g. (customer_id) for the customers table…

Interleaved tables are best for 1:1 relationships.

Re: Building CockroachDB on top of RocksDB

#18

Post by polskibus »

manigandham wrote: post #8
polskibus wrote: post #6

Earlier quoted context omitted.

I know there are many techniques that used together give good performance (optimal memory layout, compression, vectorization, etc. etc.), however I'd like to use a package that does a lot of it, same what RocksDB (or SQLite) does for OLTP cases. Is there something like that? If not, what's out there that gives the best foundation for building OLAP functionalities on top of it?

That's what Apache Arrow is, you had the right choice. That solves the processing component and you can use any number of on-disk formats like Parquet and ORC. And the hybrid of OLAP + OLTP is usually called HTAP.

Is it possible to insert new tuples to arrow model without rebuilding it from scratch?

Re: Building CockroachDB on top of RocksDB

#19

Post by perfmode »

ryanworl wrote: post #16
perfmode wrote: post #13

When a SQL implementation is built on a KV storage engine, how do tables, rows, and columns typically map to the underlying KV data model?

The CockroachDB blog post on this topic is a good summary. There is an additional trick that isn't directly KV related, but is important in a distributed environment when using a KV storage engine. When defining a hierarchy of tables, such as customers -> orders -> order_line_items, you can make the primary key of the child tables contain the primary key of the parent table. e.g. (customer_id) for the customers table…

Can you link to the post you’re referring to?

Re: Building CockroachDB on top of RocksDB

#20

Post by ryanworl »

gigatexal wrote: post #17
ryanworl wrote: post #16

Earlier quoted context omitted.

The CockroachDB blog post on this topic is a good summary. There is an additional trick that isn't directly KV related, but is important in a distributed environment when using a KV storage engine. When defining a hierarchy of tables, such as customers -> orders -> order_line_items, you can make the primary key of the child tables contain the primary key of the parent table. e.g. (customer_id) for the customers table…

Interleaved tables are best for 1:1 relationships.

Hard disagree. This is the only way in a distributed, sorted KV store to get any semblance of data locality. If Cockroach and Spanner didn't do this, they would constantly be doing 2PC for modifying data that is related but stored on different groups of machines.
Post reply on HN