Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Druid: fast column-oriented distributed data store

druid.io

Post reply on HN
Print view View original

31–40 of 50 posts

Re: Druid: fast column-oriented distributed data store

#31

Post by vog »

feral wrote: post #10
tschellenbach wrote: post #4

Druid is quickly becoming the leading open source solution for building highly scalable analytics. We evaluated it for getstream.io. Unfortunately the setup and maintenance is still very labour intensive. For startups that's a concern. Many larger companies we spoke to were extremely happy about running Druid in production though.

Second that experience: When I was at Optimizely, my team chose Druid for a large-scale analytics application, after a pretty extensive benchmarking. It was very impressive, though not trivial to set up. Blog post with a bit more detail: https://medium.com/engineers-optimizely/slicing-and-dicing-d...

I'm a bit surprised about forking the druid chef cookbook. From the blog post:

> we were delighted to find an existing community cookbook, chef-druid, that can configure and deploy a druid cluster. However, that community cookbook hasn’t been updated for nearly a year and does not support the latest version of druid. We have therefore forked off our own version, optimizely/chef-druid, which supports the latest version of druid.

Why didn't you create a pull request to the upstream project? That way, all people who are using the original cookbook could have directly benefit from your improvements, without having to discover the new fork.

Re: Druid: fast column-oriented distributed data store

#33

Post by gglanzani »

mrweasel wrote: post #32

What's the advantage of a "column-oriented" data store/database?

Analytics like sum(column) are much faster if data is saved column-oriented. Also every time you don't do select *, but select a, b, it's much easier on disk because you're not even touching some columns.

Re: Druid: fast column-oriented distributed data store

#34

Post by RyanHamilton »

tschellenbach wrote: post #4

Druid is quickly becoming the leading open source solution for building highly scalable analytics. We evaluated it for getstream.io. Unfortunately the setup and maintenance is still very labour intensive. For startups that's a concern. Many larger companies we spoke to were extremely happy about running Druid in production though.

"becoming the leading open source solution" is something you can say that doesn't mean much and is difficult to prove or disprove. There are a number of entrenched players, open source and commercial that provide a column oriented database: http://www.timestored.com/time-series-data/column-oriented-d...

That's a short description of 12 alternatives I wrote up. Druid does look interesting, it appears to have got the architecture pretty spot on and I'm going to look into it more.

Re: Druid: fast column-oriented distributed data store

#35

Post by ci5er »

eva1984 wrote: post #29
tschellenbach wrote: post #4

Druid is quickly becoming the leading open source solution for building highly scalable analytics. We evaluated it for getstream.io. Unfortunately the setup and maintenance is still very labour intensive. For startups that's a concern. Many larger companies we spoke to were extremely happy about running Druid in production though.

No it isn't Elasticsearch and Redshift are.

Huh? GP will make an unsubstantiated claim and you'll double down with an unsubstantiated counter claim (in one line!)

Seriously: What, specifically, do you disagree with, in the context of your use case, and what is your use-case?

Re: Druid: fast column-oriented distributed data store

#36

Post by thecatspaw »

brightball wrote: post #24
brightball wrote: post #20

Earlier quoted context omitted.

It takes only what it needs to survive...

We're down voting Spaceballs references? Who are you monsters!?

people who come here for serious discussions. Reddit is more the place for memes, references and stuff like that, not HN

Re: Druid: fast column-oriented distributed data store

#37

Post by mau »

tschellenbach wrote: post #4

Druid is quickly becoming the leading open source solution for building highly scalable analytics. We evaluated it for getstream.io. Unfortunately the setup and maintenance is still very labour intensive. For startups that's a concern. Many larger companies we spoke to were extremely happy about running Druid in production though.

Much of the effort for the next release of Druid (0.9) has been put into docs. Even with that you still need to handle a complex cluster made of several services: zookeeper, druid overlord, druid middlemanager, druid broker, etc... still it sounds more complicate than it actually is. What was the alternative solution adopted by your company?

Re: Druid: fast column-oriented distributed data store

#38

Post by jnordwick »

slachterman wrote: post #12
jnordwick wrote: post #8

Earlier quoted context omitted.

I mean like give me the preceding row from the current. Very useful for calculating anything from a simple rate of change to a properly doing a temporal database. Arrays as a column type doesn't get you that.

Impala supports LAG and LEAD, etc.: http://www.cloudera.com/documentation/archive/impala/2-x/2-0...

So do most databased, but they are SLOOOOOOOW.

Re: Druid: fast column-oriented distributed data store

#39

Post by jnordwick »

teraflop wrote: post #11
jnordwick wrote: post #8

Earlier quoted context omitted.

I mean like give me the preceding row from the current. Very useful for calculating anything from a simple rate of change to a properly doing a temporal database. Arrays as a column type doesn't get you that.

No, but window functions do, unless I'm not understanding what your goal is. http://www.postgresql.org/docs/current/static/functions-wind...

I actually was using that yesterday. So so so so very very very very slow.

Re: Druid: fast column-oriented distributed data store

#40

Post by Exuma »

Our 2-man team set up Druid........ i took 5+ months and was excruciating to configure and get running smoothly (things were slightly more complicated because we decided to use docker). It also took ~30 servers to make a truly fault-tolerant setup.

With that said, it works very well, but it definitely came at the cost of a good dose of sanity.

Post reply on HN