How should you build a high-performance column store for the 2020s?

> I assume that the Arrow developers will eventually read my 2006 paper on compression in column-stores and expand their compression options to include other schemes which can be operated on directly (such as run-length-encoding and bit-vector compression).

In this blog post, I don’t agree with:

> Almost every single major data-processing platform that has emerged in the last decade has been either open source.

That’s somewhat true by definition. OTOH, I also know most financial firms use proprietary solutions (which leverage open source components).

put data files in directory

name data files after columns

use ".data" filename extension for data files

write a tool to create index files (append ".index" to the name of the input text file) that map record number to byte offset in data file

If data files are all Each index file is a packed array of 32 bit integers

Write a tool to create length files ".length" that count the number of entries in a data file

Generate .length files for all data files

Use mmap to access index files

Use C for all of the above

This is for variable-length data values. Not every column will have these, making the .index files redundant in this case; the .index files should not be created in this case and program logic should support both uniform value length access and nonuniform value length access. The reason to prefer two access modes is to keep data from the .index files out of the cache when it is redundant.

When all of this is done, the next thing to do is write a tool to test the cache characteristics on your processor by implementing sorting algorithms and testing their performance. Unless you are using a GPU (why?) all data your algorithm touches will go through every level of the cache hierarchy, forcing other data out. If possible, use a tool that reports hardware diagnostics. These tools may be provided by the processor vendor.

Now, there is a trend to give the programmer control over cache behavior

https://stackoverflow.com/questions/9544094/how-to-mark-some...

I don't know if this is worth exploring or a wild goose chase. It may improve performance for some tasks, but it sounds a little strange for the programmer to tell the computer how to use the cache...shouldn't the operating system do this?

Anyway, that's a start.

Once you're both distributed and consistent, the problems today's stores are architected around, go away. Your distributed queries can index the immutable log however they like. column-oriented, row-oriented, documents, time-oriented, graphs, immutability means you can do all of it, as a library in your application process

http://www.datomic.com/ - it's what you get when Facebook's graph datastore has a baby with immutability.

If Apache Arrow takes off I hope BigQuery will support it as a data interchange format in the future. Zero-copy is pretty awesome, as are open standards in general. This feature does not exist in BigQuery today (as far as I know - definitely not as discussed in the source).