How should you build a high-performance column store for the 2020s?

Post by dustingetz » Tue, Nov 14, 2017, 3:05 PM UTC

elvinyung wrote: post #69

dustingetz wrote: post #66

Earlier quoted context omitted.

You actually don't need to have the same storage data layout if you use a time series as a starting point; because you can maintain different data layouts in parallel, and the time dimension permits strong consistency across them all. If this is what you mean by a "database frontend", I am really confused as to why you object to this? I think this property of time series is going to prove very important in the 2020s

Again, I don't care about distributed consistency here, nor is it mutually exclusive with what I'm talking about. The question we're trying to answer is whether there exists at all a storage engine, at the scope of a single node (obviously generalizable/scalable), that can fit all use cases well. Once you figure out the answer to that, obviously having one storage engine is simpler than having two storage engines.

Post by dustingetz » Thu, Nov 16, 2017, 6:24 PM UTC

elvinyung wrote: post #72

dustingetz wrote: post #71

Earlier quoted context omitted.

Parallel but consistent indexes could be done in a single process too. The tradeoff is natural reads at the cost of extra constant factor of write computation and storage. Which is exactly the tradeoff RDBMS indexes already make.

That's still not what I'm talking about. https://cs.brown.edu/~ugur/fits_all.pdf