Columnar Storage

#1

Post by mattyb » Thu, Jan 31, 2013, 3:54 AM UTC

Re: Columnar Storage

#2

Post by nwatson » Thu, Jan 31, 2013, 4:37 AM UTC

I worked on a columnar database engine deployed on commodity Linux clusters in 2001, initial release 2002. This was at Sensage. We built a very fast DB engine that could store log data with 40x compression over traditional RDBMSs and ran log queries much more quickly than commercial DBs could at the time. I think we were too early.

Current Sensage company blurb about the event-data warehouse: http://sensage.com/content/clustered-columnar-database and http://sensage.com/content/why-columnar%E2%80%A6not-row-base...

Patent work: http://www.patentgenius.com/patent/7024414.html

The core engineering team was CTO + 3 engineers. Best engineering experience of my life. I wasn't involved at the lowest DB storage level, the guys who did that did a great job.

Michael Stonebraker, technical advisor to Sensage, learned from the Sensage mistakes and built Vertica.

Re: Columnar Storage

#3

Post by vinkelhake » Thu, Jan 31, 2013, 4:45 AM UTC

Google's Dremel[0] also belong in this category.

[0] http://research.google.com/pubs/pub36632.html

Re: Columnar Storage

#4

Post by hkmurakami » Thu, Jan 31, 2013, 5:02 AM UTC

When I hear columnar storage, I can't help but remember all the stories my friends have told me about kx systems (I'd love to share them but my memory is so fuzzy that I wouldn't be doing them justice. Hopefully some of the HNers here can share some of theirs though!) [1][2].

[1] http://kx.com/

[2] http://en.wikipedia.org/wiki/K_(programming_language)

Re: Columnar Storage

#5

Post by clouderashiring » Thu, Jan 31, 2013, 5:27 AM UTC

nwatson wrote: post #2

I worked on a columnar database engine deployed on commodity Linux clusters in 2001, initial release 2002. This was at Sensage. We built a very fast DB engine that could store log data with 40x compression over traditional RDBMSs and ran log queries much more quickly than commercial DBs could at the time. I think we were too early. Current Sensage company blurb about the event-data warehouse: http://sensage.com/conte…

Sounds like Impala could use your talent. You should talk with them.

Re: Columnar Storage

#6

Post by toast0 » Thu, Jan 31, 2013, 5:27 AM UTC

possibly dumb question; what's the difference between columnar storage and a bunch of row-major tables with an id column, and one column of data?

Re: Columnar Storage

#7

Post by nwatson » Thu, Jan 31, 2013, 5:39 AM UTC

clouderashiring wrote: post #5

nwatson wrote: post #2

I worked on a columnar database engine deployed on commodity Linux clusters in 2001, initial release 2002. This was at Sensage. We built a very fast DB engine that could store log data with 40x compression over traditional RDBMSs and ran log queries much more quickly than commercial DBs could at the time. I think we were too early. Current Sensage company blurb about the event-data warehouse: http://sensage.com/conte…

Sounds like Impala could use your talent. You should talk with them.

Thanks for the pointer! Looking at Cloudera's jobs board, most tech-heavy work is in SF or Palo Alto. I live in San Mateo, convenient to both of those, but ... I'm likely leaving for the North Carolina RTP area soon.

Re: Columnar Storage

#8

Post by gruseom » Thu, Jan 31, 2013, 5:52 AM UTC

hkmurakami wrote: post #4

When I hear columnar storage, I can't help but remember all the stories my friends have told me about kx systems (I'd love to share them but my memory is so fuzzy that I wouldn't be doing them justice. Hopefully some of the HNers here can share some of theirs though!) [1][2]. [1] http://kx.com/ [2] http://en.wikipedia.org/wiki/K_(programming_language)

Get your friends to join HN and tell us?

Re: Columnar Storage

#9

Post by grundprinzip » Thu, Jan 31, 2013, 6:03 AM UTC

nwatson wrote: post #2

I worked on a columnar database engine deployed on commodity Linux clusters in 2001, initial release 2002. This was at Sensage. We built a very fast DB engine that could store log data with 40x compression over traditional RDBMSs and ran log queries much more quickly than commercial DBs could at the time. I think we were too early. Current Sensage company blurb about the event-data warehouse: http://sensage.com/conte…

Just one point: Vertica came out of the research project C-Store which was a PhD thesis by Dan Abadi. However, the outcome is the same :)

Re: Columnar Storage

#10

Post by grundprinzip » Thu, Jan 31, 2013, 6:07 AM UTC

toast0 wrote: post #6

possibly dumb question; what's the difference between columnar storage and a bunch of row-major tables with an id column, and one column of data?

The biggest advantage of a pure column-oriented DBMS comes from having the positional information indirectly available without the requirement to store this ID information. During query execution the required position lists can then be generated.

In addition. If you considere the record format of traditional row-oriented databases you will see that the overhead of storing a single attribute record is rather high. Since with column-oriented DBMS its all about IO performance (Disk/Memory, Memory/CPU) such overhead can diminish the advantage.

Thus typical column stores tend to use only single strings of sequential memory to store the data. This can even be enhanced by applying dictionary compression and as a result only storing integer values. And modern CPUs are good in processing lots of them.