Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Druid: fast column-oriented distributed data store

druid.io

Post reply on HN
Print view View original

21–30 of 50 posts

Re: Druid: fast column-oriented distributed data store

#21

Post by joefkelley »

fangjin wrote: post #19
joefkelley wrote: post #18

Earlier quoted context omitted.

If one of these databases included a really good SparkSQL connector (including great predicate push-down), I could see that being hugely beneficial to growth. Part of why people are using these systems is because you can connect in many different types of data. Make your database part of that ecosystem and you can gradually take in more of the analytics workload without the users losing the ability to also throw in s…

Druid has a SparkSQL connector: https://github.com/SparklineData/spark-druid-olap

Cool! Yeah I poked around the druid site and didn't find anything originally, but this looks pretty promising. It's hard to tell how full-featured it is without getting hands-on, but I see something like this making Druid much more usable in a lot of analytics environments.

Re: Druid: fast column-oriented distributed data store

#22

Post by threeseed »

joefkelley wrote: post #18
threeseed wrote: post #14

Earlier quoted context omitted.

Druid isn't even close to being the leading solution. The overwhelming majority of places doing big data analytics will be Hadoop/Spark using core HDFS. Then most places will augment this with a range of database solutions depending on how structured/clean the data is and the various workloads. Cassandra, HBase, MongoDB, Teradata, Oracle, ElasticSearch, Greenplum are all pretty common place in most enterprises. And o…

If one of these databases included a really good SparkSQL connector (including great predicate push-down), I could see that being hugely beneficial to growth. Part of why people are using these systems is because you can connect in many different types of data. Make your database part of that ecosystem and you can gradually take in more of the analytics workload without the users losing the ability to also throw in s…

Cassandra, MongoDB and ElasticSearch all have SparkSQL DataSource drivers with predicate pushdown support. Most of the limitations in that support is from the database itself and not from the driver.

The biggest problems with SparkSQL is simply in its limited support for ANSI SQL. It's getting better with every release but not nearly quick enough.

Re: Druid: fast column-oriented distributed data store

#23

Post by NamTaf »

The realtime ingestion is interesting especially if I can still batch import. When processing machine data, I've found that a quantity of sources come in chunks (logfiles written out every 24 hours for exmaple) but the eventual aim is to migrate to realtime (i.e.: a data point every n seconds/minutes/etc. where you instantly consume that data point) streaming.

If this transition is easy without reworking infrastructure, the solution is far more attractive.

Re: Druid: fast column-oriented distributed data store

#26

Post by deepanchor »

fangjin wrote: post #17
threeseed wrote: post #14

Earlier quoted context omitted.

Druid isn't even close to being the leading solution. The overwhelming majority of places doing big data analytics will be Hadoop/Spark using core HDFS. Then most places will augment this with a range of database solutions depending on how structured/clean the data is and the various workloads. Cassandra, HBase, MongoDB, Teradata, Oracle, ElasticSearch, Greenplum are all pretty common place in most enterprises. And o…

Druid's main value add to the data infrastructure space is around power user-facing data applications at scale. The queries it is best at are OLAP/business intelligence style queries. It isn't really designed to be a general processing tool such as Hadoop or Spark. The open source data space is very complex, and there are many different solutions targeted towards many different use cases. Druid is better than other s…

As someone who hasn't yet had the opportunity to use many of these systems, this was a great high-level overview of how the various systems fit together. Thanks for writing it!

Re: Druid: fast column-oriented distributed data store

#29

Post by eva1984 »

tschellenbach wrote: post #4

Druid is quickly becoming the leading open source solution for building highly scalable analytics. We evaluated it for getstream.io. Unfortunately the setup and maintenance is still very labour intensive. For startups that's a concern. Many larger companies we spoke to were extremely happy about running Druid in production though.

No it isn't

Elasticsearch and Redshift are.

Re: Druid: fast column-oriented distributed data store

#30

Post by ddorian43 »

whitegrape wrote: post #16

Has anyone done a meaningful private benchmark comparison with http://www.scylladb.com/ ? I didn't find one online.

It is a 100% different type of database. Druid is olap while scylladb is oltp. They have nothing in common (except for the "columnar" name)
Post reply on HN