Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Druid: A Real-Time Analytical Data Store

micahlerner.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Druid: A Real-Time Analytical Data Store

#21

Post by drewbug01 »

Plough_Jogger wrote: post #2

My sense from the few projects I've seen attempt to use Druid is that there is quite a lot of infrastructure overhead / DevOps support required to manage a cluster at scale, and that fairly complex ingestion pipelines are required to load the data in the right format. Anecdotally, I've heard that ClickHouse is easier to deploy from this perspective with similar performance, but would love to get others views / experi…

Clickhouse benefits from the ability to get started as a single binary (and even includes a clickhouse-local binary to do ad-hoc analysis on CSVs on your laptop). There’s only one node type, as well. It’s simpler and easier in that sense.

Running it at scale is different. It includes everything you need, and it’s not horrible of course - but there are certainly a lot of sharp edges to be mindful of.

Re: Druid: A Real-Time Analytical Data Store

#24

Post by msarrel »

imdsm wrote: post #13

What other options are there of this type? ClickHouse — anything else?

Clickhouse, trino, kylin, bigquery, snowflake, plenty of competitors.

Having worked in the space for a few years, druid is very fast when you know exactly what you're doing. Until you reach that point, you're going to have a bad time. And if you try to run it yourself under kubernetes you're going to have a really bad bad time. Druid is amazingly fast, but no development effort is going into making it easier to run.

Re: Druid: A Real-Time Analytical Data Store

#25

Post by kpfly2022 »

david38 wrote: post #4
Plough_Jogger wrote: post #2

My sense from the few projects I've seen attempt to use Druid is that there is quite a lot of infrastructure overhead / DevOps support required to manage a cluster at scale, and that fairly complex ingestion pipelines are required to load the data in the right format. Anecdotally, I've heard that ClickHouse is easier to deploy from this perspective with similar performance, but would love to get others views / experi…

I run a ClickHouse cluster. I’ve heard Druid is more difficult, but ClickHouse isn’t exactly a piece of cake. It’s great for a while, but sometimes you can get bitten by weird states. Still, compared to everything I’ve used, for the use case where it excels, it really excels.

clickhouse is difficult to operate and does support join well, i use https://github.com/apache/doris replace our clickhouse and druid workloads

Re: Druid: A Real-Time Analytical Data Store

#26

Post by mikehollinger »

imdsm wrote: post #13

What other options are there of this type? ClickHouse — anything else?

We evaluated pulsar plus trino/presto a bit ago which seems to be similar in spirit. Pulsar promised to be a slick follow on to Kafka with tiered storage. We found some terrible write/read latency problems when we tried to force the spillover to s3. We gave up on it.

Re: Druid: A Real-Time Analytical Data Store

#27

Post by dominotw »

hobofan wrote: post #20
didip wrote: post #14

Earlier quoted context omitted.

We found the opposite. Setting up Druid clusters is the easiest compared to its competitors: Clickhouse and Pinot. Especially since ingestion goes straight to S3. We don’t really worry about backups (just deal with PG backups). Just make sure your ZK is happy and all will be well. The hard part about Druid is tuning: - the ingestions: Spec definition, compaction, sharding strategy, RAM consumptions, etc. - and query…

> Just make sure your ZK is happy and all will be well. That sounds like the opposite of easy to setup (and maintain).

We had druid in production and this was our main weak link. Its really hire to find people who know how to operate ZK well in production.

Re: Druid: A Real-Time Analytical Data Store

#30

Post by dominotw »

msarrel wrote: post #24
imdsm wrote: post #13

What other options are there of this type? ClickHouse — anything else?

Clickhouse, trino, kylin, bigquery, snowflake, plenty of competitors. Having worked in the space for a few years, druid is very fast when you know exactly what you're doing. Until you reach that point, you're going to have a bad time. And if you try to run it yourself under kubernetes you're going to have a really bad bad time. Druid is amazingly fast, but no development effort is going into making it easier to run.

This has exactly been my experience. It needs you know exactly what you are doing and what you are using the system for.

Suprised to see comments on this thread saying they are running druid on kubernetes with no issues at all.

Post reply on HN