Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Druid: fast column-oriented distributed data store

druid.io

Post reply on HN
Print view View original

11–20 of 50 posts

Re: Druid: fast column-oriented distributed data store

#11

Post by teraflop »

jnordwick wrote: post #8
ktamura wrote: post #6

Earlier quoted context omitted.

This is simply false. PostgreSQL + cstore_ftw, Presto, Impala, etc. all support "array oriented ability" via window functions.

I mean like give me the preceding row from the current. Very useful for calculating anything from a simple rate of change to a properly doing a temporal database. Arrays as a column type doesn't get you that.

No, but window functions do, unless I'm not understanding what your goal is.

http://www.postgresql.org/docs/current/static/functions-wind...

Re: Druid: fast column-oriented distributed data store

#12

Post by slachterman »

jnordwick wrote: post #8
ktamura wrote: post #6

Earlier quoted context omitted.

This is simply false. PostgreSQL + cstore_ftw, Presto, Impala, etc. all support "array oriented ability" via window functions.

I mean like give me the preceding row from the current. Very useful for calculating anything from a simple rate of change to a properly doing a temporal database. Arrays as a column type doesn't get you that.

Impala supports LAG and LEAD, etc.: http://www.cloudera.com/documentation/archive/impala/2-x/2-0...

Re: Druid: fast column-oriented distributed data store

#13

Post by fangjin »

feral wrote: post #10
tschellenbach wrote: post #4

Druid is quickly becoming the leading open source solution for building highly scalable analytics. We evaluated it for getstream.io. Unfortunately the setup and maintenance is still very labour intensive. For startups that's a concern. Many larger companies we spoke to were extremely happy about running Druid in production though.

Second that experience: When I was at Optimizely, my team chose Druid for a large-scale analytics application, after a pretty extensive benchmarking. It was very impressive, though not trivial to set up. Blog post with a bit more detail: https://medium.com/engineers-optimizely/slicing-and-dicing-d...

Druid committer here. We spent a lot of time in the early days on making sure the system worked (at scale) and now we're spending more time to make it much easier to set up and manage.

Re: Druid: fast column-oriented distributed data store

#14

Post by threeseed »

tschellenbach wrote: post #4

Druid is quickly becoming the leading open source solution for building highly scalable analytics. We evaluated it for getstream.io. Unfortunately the setup and maintenance is still very labour intensive. For startups that's a concern. Many larger companies we spoke to were extremely happy about running Druid in production though.

Druid isn't even close to being the leading solution. The overwhelming majority of places doing big data analytics will be Hadoop/Spark using core HDFS.

Then most places will augment this with a range of database solutions depending on how structured/clean the data is and the various workloads. Cassandra, HBase, MongoDB, Teradata, Oracle, ElasticSearch, Greenplum are all pretty common place in most enterprises.

And of course the ridiculous array of SQL engines on top of HDFS/S3/whatever else e.g. Hive, Spark SQL, Presto, Drill, SAP etc.

Re: Druid: fast column-oriented distributed data store

#15

Post by pookeh »

fangjin wrote: post #13
feral wrote: post #10

Earlier quoted context omitted.

Second that experience: When I was at Optimizely, my team chose Druid for a large-scale analytics application, after a pretty extensive benchmarking. It was very impressive, though not trivial to set up. Blog post with a bit more detail: https://medium.com/engineers-optimizely/slicing-and-dicing-d...

Druid committer here. We spent a lot of time in the early days on making sure the system worked (at scale) and now we're spending more time to make it much easier to set up and manage.

It might be helpful perhaps to provide some kubernetes configuration set up (or like a presetup kubernetes running in vagrant) that has all the nodes correctly configured out of the box to easily get started with development and prototyping.

Re: Druid: fast column-oriented distributed data store

#17

Post by fangjin »

threeseed wrote: post #14
tschellenbach wrote: post #4

Druid is quickly becoming the leading open source solution for building highly scalable analytics. We evaluated it for getstream.io. Unfortunately the setup and maintenance is still very labour intensive. For startups that's a concern. Many larger companies we spoke to were extremely happy about running Druid in production though.

Druid isn't even close to being the leading solution. The overwhelming majority of places doing big data analytics will be Hadoop/Spark using core HDFS. Then most places will augment this with a range of database solutions depending on how structured/clean the data is and the various workloads. Cassandra, HBase, MongoDB, Teradata, Oracle, ElasticSearch, Greenplum are all pretty common place in most enterprises. And o…

Druid's main value add to the data infrastructure space is around power user-facing data applications at scale. The queries it is best at are OLAP/business intelligence style queries. It isn't really designed to be a general processing tool such as Hadoop or Spark. The open source data space is very complex, and there are many different solutions targeted towards many different use cases. Druid is better than other solutions at some of these use cases, and worse than other solutions at others.

I wrote my interpretation of the current open source data landscape here for anyone interested: http://imply.io/post/2015/11/04/big-data-zoo.html

Re: Druid: fast column-oriented distributed data store

#18

Post by joefkelley »

threeseed wrote: post #14
tschellenbach wrote: post #4

Druid is quickly becoming the leading open source solution for building highly scalable analytics. We evaluated it for getstream.io. Unfortunately the setup and maintenance is still very labour intensive. For startups that's a concern. Many larger companies we spoke to were extremely happy about running Druid in production though.

Druid isn't even close to being the leading solution. The overwhelming majority of places doing big data analytics will be Hadoop/Spark using core HDFS. Then most places will augment this with a range of database solutions depending on how structured/clean the data is and the various workloads. Cassandra, HBase, MongoDB, Teradata, Oracle, ElasticSearch, Greenplum are all pretty common place in most enterprises. And o…

If one of these databases included a really good SparkSQL connector (including great predicate push-down), I could see that being hugely beneficial to growth. Part of why people are using these systems is because you can connect in many different types of data. Make your database part of that ecosystem and you can gradually take in more of the analytics workload without the users losing the ability to also throw in some plain old text files or something and join between multiple worlds.

Re: Druid: fast column-oriented distributed data store

#19

Post by fangjin »

joefkelley wrote: post #18
threeseed wrote: post #14

Earlier quoted context omitted.

Druid isn't even close to being the leading solution. The overwhelming majority of places doing big data analytics will be Hadoop/Spark using core HDFS. Then most places will augment this with a range of database solutions depending on how structured/clean the data is and the various workloads. Cassandra, HBase, MongoDB, Teradata, Oracle, ElasticSearch, Greenplum are all pretty common place in most enterprises. And o…

If one of these databases included a really good SparkSQL connector (including great predicate push-down), I could see that being hugely beneficial to growth. Part of why people are using these systems is because you can connect in many different types of data. Make your database part of that ecosystem and you can gradually take in more of the analytics workload without the users losing the ability to also throw in s…

Druid has a SparkSQL connector: https://github.com/SparklineData/spark-druid-olap
Post reply on HN