Possibly naive question, but isn't an index (in a classical relational database) the same as a column store?
How should you build a high-performance column store for the 2020s?
31–40 of 73 posts
Re: How should you build a high-performance column store for the 2020s?
#32Are column-store databases relevant on SSD/NVME?
I ask because on a physical medium like hard disk, storing data on physical disk in column orientation can make a significant improvement to read operations.
But with SSD/NVME, you don’t have to worry about the inherent slowness of physical platters that exist in hard disk.
Re: How should you build a high-performance column store for the 2020s?
#33Are column-store databases relevant on SSD/NVME? I ask because on a physical medium like hard disk, storing data on physical disk in column orientation can make a significant improvement to read operations. But with SSD/NVME, you don’t have to worry about the inherent slowness of physical platters that exist in hard disk.
Columnar stores are as much about the compression benefits as the physical layout on disk. The article goes through a bunch of different relevant compression strategies.
Re: How should you build a high-performance column store for the 2020s?
#34Datastore of 2020s will be designed around an immutable log because it permits both strong consistency and horizontal scaling (like git). Once you're both distributed and consistent, the problems today's stores are architected around, go away. Your distributed queries can index the immutable log however they like. column-oriented, row-oriented, documents, time-oriented, graphs, immutability means you can do all of it…
All datastores already have WAL logging which is effectively the same, and commonly used for replication, changefeeds and other downstream consumers. Saving the entire history (with compaction) and some CQRS patterns is nothing new.
At any decent scale, most companies now just use a dedicated log like Kafka or Pulsar as the main backbone to support more flexibility in producers and consumers. Either way, none of this has to do with column-stores as the actual representation of data.
Re: How should you build a high-performance column store for the 2020s?
#35Possibly naive question, but isn't an index (in a classical relational database) the same as a column store?
An index stores pointers to rows based on the column value; the values are still stored as rows though.
So when you query on an indexed column, you'll have contiguous access on the index, but the rows themselves may be stored on disk based on a different column (so you'll could get the row pointers for a range query in one go from the index but fetching the rows would be random lookups). But if you want to view an entire row, its trivial, because the full row data is contiguous on disk.
Column store groups the table values by the column, so the values of col A will be contigous, and col B will be contiguous, (but not pairwise!) but if you want to view the entire row, you'll probably have to do 2 lookups in random locations. But the range query on col A, selecting only col A, becomes a trivial fetch.
Thats my understanding anyways
Re: How should you build a high-performance column store for the 2020s?
#36Are column-store databases relevant on SSD/NVME? I ask because on a physical medium like hard disk, storing data on physical disk in column orientation can make a significant improvement to read operations. But with SSD/NVME, you don’t have to worry about the inherent slowness of physical platters that exist in hard disk.
Even SSD's are super slow compared to RAM. When you want to read a few bytes from millions of rows, an SSD has to decode an entire block of data for every read.
Also, even with NVMe SSD's, there is a lot of operating system overhead associated with every read. Having layers of drivers to orchestrate the transfer of 2 bytes of data you wanted really slows it down...
Re: How should you build a high-performance column store for the 2020s?
#37Most of these techniques are already in production:
Microsoft SQL Server has columnstore indexes and can even be combined with its in-memory tables. MemSQL has been doing this for years and v6 is incredibly fast, also combines in-memory row-stores. ClickHouse is very good if you don't mind more operations work. MariaDB has the ColumnStore storage engine, Postgre has the cstore_fdw extension. Vertica, Greenplum, Druid, etc. EventQL was an interesting project but abandoned now.
AWS RedShift, Azure SQL Data Warehouse, Snowflake Data, Google BigQuery are the hosted options, with BQ being the most advanced with its vertical integration.
If you want to operationalize Apache Arrow today, Dremio is built around it and works similar to Apache Drill and Spark to run distributed queries and joins across data sources.
Re: How should you build a high-performance column store for the 2020s?
#38I wonder if the 2020s column store would outperform kdb, which was written in the 1990s with a UI from the 1950s.
Re: How should you build a high-performance column store for the 2020s?
#39This already exists, in Google BigQuery. Uses darn near every trick in the book, and some that aren’t in the book. Source: shipped it.
It is frustrating that google is always 5 to 10 years ahead of everyone else but they never open source their back-end technologies (except recently with the ML stuff). The whole reason hadoop exists is because google only released whitepapers (which was good) but not code. I wonder whether google really benefits by this strategy, given that they have to be an ecosystem instead of benefitting from being a part of one. I also wonder whether the industry is better off by having to cooperatively reinvent the google architecture. I doubt the hadoop ecosystem would have arisen had it been google code at the heart.
Re: How should you build a high-performance column store for the 2020s?
#40Possibly naive question, but isn't an index (in a classical relational database) the same as a column store?
An index for a column-store and row-store is (conceptually) the same. Why is different is
1. How data is represented. A database is a collection of column objects. For example, it is easy to create a column or delete a column.
2. How data is being processed (queried). The engine processes columns as objected managed by the system