Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How should you build a high-performance column store for the 2020s?

lemire.me

Post reply on HN
Print view View original

51–60 of 73 posts

Re: How should you build a high-performance column store for the 2020s?

#51

Post by beaumayns »

geocar wrote: post #43
bboreham wrote: post #38

I wonder if the 2020s column store would outperform kdb, which was written in the 1990s with a UI from the 1950s.

Unlikely. Two reasons are at the top of my mind: 1. The current best efforts in benchmarking are focusing on queries that "look" similar, and yet kdb is still 400x faster than Hadoop for those queries. For example: select avg size by sym,time.hh from trade where date=d,sym in S SELECT sym,HOUR(time),AVG(size) FROM trade NATURAL JOIN s WHERE date=d GROUP BY sym,HOUR(time); To answer this question, the database has to…

kdb also allows you to do much more than SQL, since a select is more or less just syntactic sugar for a certain set of operations on columns as arrays. The full language is available in the context of the query, or you could just treat your table as a bunch of arrays in the context of a larger program.

It's a really elegant way of dealing with large amounts of data, although the downside is that you've typically got to build a lot of the nice-to-have dbms type infrastructure yourself.

It would be interesting if BigQuery or Redshift ever figure out that they could have a much more powerful system if they stuck an array language on the front of their storage engines.

Re: How should you build a high-performance column store for the 2020s?

#52

Post by mikecb »

Joeri wrote: post #39
xoogler_thr wrote: post #2

This already exists, in Google BigQuery. Uses darn near every trick in the book, and some that aren’t in the book. Source: shipped it.

It is frustrating that google is always 5 to 10 years ahead of everyone else but they never open source their back-end technologies (except recently with the ML stuff). The whole reason hadoop exists is because google only released whitepapers (which was good) but not code. I wonder whether google really benefits by this strategy, given that they have to be an ecosystem instead of benefitting from being a part of one…

Apache Beam, gRPC/protobuf, Kubernetes. There are examples besides Tensorflow.

Re: How should you build a high-performance column store for the 2020s?

#53

Post by didibus »

dustingetz wrote: post #49
manigandham wrote: post #34

Earlier quoted context omitted.

All datastores already have WAL logging which is effectively the same, and commonly used for replication, changefeeds and other downstream consumers. Saving the entire history (with compaction) and some CQRS patterns is nothing new. At any decent scale, most companies now just use a dedicated log like Kafka or Pulsar as the main backbone to support more flexibility in producers and consumers. Either way, none of this…

It's definitely not new, but it is innovative. Kafka can totally be an implementation detail of a system like what we are discussing. Once you're immutable, we're no longer constrained to a single "actual representation of data"; you can maintain many in parallel, so long as there is a way to keep the representations consistent (that time dimension is really important!) CQRS has the right fundamental constituents but…

How does datomic play with encryption? Say I wanted things encrypted at rest, but still queryable?

Re: How should you build a high-performance column store for the 2020s?

#54

Post by dozzie »

geocar wrote: post #43
bboreham wrote: post #38

I wonder if the 2020s column store would outperform kdb, which was written in the 1990s with a UI from the 1950s.

Unlikely. Two reasons are at the top of my mind: 1. The current best efforts in benchmarking are focusing on queries that "look" similar, and yet kdb is still 400x faster than Hadoop for those queries. For example: select avg size by sym,time.hh from trade where date=d,sym in S SELECT sym,HOUR(time),AVG(size) FROM trade NATURAL JOIN s WHERE date=d GROUP BY sym,HOUR(time); To answer this question, the database has to…

> it's hard to be much faster than kdb: 10 billion rows completes in 70msec on kdb, but Hadoop takes something like 30 seconds.

Indeed, it's hard for exhaustive search beat index lookup.

Re: How should you build a high-performance column store for the 2020s?

#55

Post by redditaddict »

elvinyung wrote: post #27

I think one interesting project in the near future could be to try and build a column-oriented storage engine that's "good enough" for both OLAP and OLTP workloads. The main precedent here is Spanner's Ressi storage engine, which, according to the most recent paper [1], uses a PAX-like format (with blocks arranged row-oriented, but values within a block are column-oriented, so kind of like Parquet) for on-disk data,…

"I think one interesting project in the near future could be to try and build a column-oriented storage engine that's "good enough" for both OLAP and OLTP workloads." SAP Hana is an example of a system that fits into this category. This isn't new either and has been existing since the 90s. Sybase IQ (which SAP acquired) was the first commercially successful columnar database. They have an in-memory row engine to handle OLTP. OLAP queries perform exceptionally well due to the column oriented nature of the storage. Customer deployments are in the 100s of TBs and low PBs these days. Blows most open source software in terms of performance if you are willing to shell out the $. Source: I work at SAP.

Re: How should you build a high-performance column store for the 2020s?

#56

Post by dustingetz »

noahdesu wrote: post #29
dustingetz wrote: post #21

Earlier quoted context omitted.

Both what you write and what I wrote are true. To scale writes you shard writes. This generally means running multiple databases (in any DBMS) The key insight is that Datomic can cross-query N databases as a first class concept, like a triple store. You can write a sophisticated relational query against Events and Photos and Instagram, as if they are one database (when in fact they are not). This works because Datomi…

What is the primary reason people choose Datomic? From reading the website, I get the impression that the append-only nature and time-travel features are a major selling point, but in other places its just the datalog and clojure interfaces. I'm sure it's a mix, but what brings people to the system to begin with?

Datomic is my default choice for any data processing software (which is almost everything).

Immutability in-and-of-itself is not the selling point. Consider why everyone moved from CVS/SVN to Git/dvcs. The number of people who moved to git then and said "man I really wish I could go back to SVN" is approximately zero. Immutability isn't why. Git is just better at everything, that's why. Immutability is the "how".

I don't see why I would use an RDBMS to store data ever again. It's not like I woke up one day and started architecting all my applications around time travel†. It's that a lot of the accidental complexity inherent to RDBMS - ORM, N+1 problems (batching vs caching), poorly scaling joins, pressure to denormalize to stay fast, eventual consistency at scale...

Datomic's pitch is it makes all these problems go away. Immutability is simply the how. Welcome to the 2020s.

†actually I did, my startup is http://hyperfiddle.net/

Re: How should you build a high-performance column store for the 2020s?

#57

Post by dustingetz »

didibus wrote: post #53
dustingetz wrote: post #49

Earlier quoted context omitted.

It's definitely not new, but it is innovative. Kafka can totally be an implementation detail of a system like what we are discussing. Once you're immutable, we're no longer constrained to a single "actual representation of data"; you can maintain many in parallel, so long as there is a way to keep the representations consistent (that time dimension is really important!) CQRS has the right fundamental constituents but…

How does datomic play with encryption? Say I wanted things encrypted at rest, but still queryable?

That's a question for the mailing list, but IIRC the new product "Datomic Cloud" (aws-native rewrite) does this out of the box. Datomic Cloud isn't out yet but Cognitect says 2017 Q4. https://www.youtube.com/watch?v=Ljvhjei3tWU

Re: How should you build a high-performance column store for the 2020s?

#58

Post by [deleted] »

est wrote: post #47
dustingetz wrote: post #8

Datastore of 2020s will be designed around an immutable log because it permits both strong consistency and horizontal scaling (like git). Once you're both distributed and consistent, the problems today's stores are architected around, go away. Your distributed queries can index the immutable log however they like. column-oriented, row-oriented, documents, time-oriented, graphs, immutability means you can do all of it…

Counters is the achilles' heel of immutable log based dbs.

[deleted]

Re: How should you build a high-performance column store for the 2020s?

#59

Post by noahdesu »

dustingetz wrote: post #56
noahdesu wrote: post #29

Earlier quoted context omitted.

What is the primary reason people choose Datomic? From reading the website, I get the impression that the append-only nature and time-travel features are a major selling point, but in other places its just the datalog and clojure interfaces. I'm sure it's a mix, but what brings people to the system to begin with?

Datomic is my default choice for any data processing software (which is almost everything). Immutability in-and-of-itself is not the selling point. Consider why everyone moved from CVS/SVN to Git/dvcs. The number of people who moved to git then and said "man I really wish I could go back to SVN" is approximately zero. Immutability isn't why. Git is just better at everything, that's why. Immutability is the "how". I d…

Thanks for the explanation, and I definitely agree with all your points. The reason I ask is that over the past year or so, we have been hacking on a little side project that maps a copy-on-write tree to a distributed shared log (https://nwat.io/blog/2016/08/02/introduction-to-the-zlog-tra...). This design effectively produces an append-only database with transactions (similar to the rocksdb interface), and means that you can have full read scalability for any past database snapshot. We don't have any query language running on top, but have been looking for interesting use cases.

Re: How should you build a high-performance column store for the 2020s?

#60

Post by elvinyung »

redditaddict wrote: post #55
elvinyung wrote: post #27

I think one interesting project in the near future could be to try and build a column-oriented storage engine that's "good enough" for both OLAP and OLTP workloads. The main precedent here is Spanner's Ressi storage engine, which, according to the most recent paper [1], uses a PAX-like format (with blocks arranged row-oriented, but values within a block are column-oriented, so kind of like Parquet) for on-disk data,…

"I think one interesting project in the near future could be to try and build a column-oriented storage engine that's "good enough" for both OLAP and OLTP workloads." SAP Hana is an example of a system that fits into this category. This isn't new either and has been existing since the 90s. Sybase IQ (which SAP acquired) was the first commercially successful columnar database. They have an in-memory row engine to hand…

Oh, I don't mean a database frontend that can handle both OLTP and OLAP workloads, usually by having some kind of OLAP column-store and some kind of OLTP main memory row-store. I know there's a lot of those (not only HANA, but also MemSQL, SQL Server, etc.)

The fun thing to try and imagine here is having literally the same physical data format that works for both kinds of workloads.

Post reply on HN