Druid: A Real-Time Analytical Data Store
micahlerner.com
Druid: A Real-Time Analytical Data Store
1–10 of 30 posts
Re: Druid: A Real-Time Analytical Data Store
#2My sense from the few projects I've seen attempt to use Druid is that there is quite a lot of infrastructure overhead / DevOps support required to manage a cluster at scale, and that fairly complex ingestion pipelines are required to load the data in the right format.
Anecdotally, I've heard that ClickHouse is easier to deploy from this perspective with similar performance, but would love to get others views / experience with these and similar data stores.
Re: Druid: A Real-Time Analytical Data Store
#3My sense from the few projects I've seen attempt to use Druid is that there is quite a lot of infrastructure overhead / DevOps support required to manage a cluster at scale, and that fairly complex ingestion pipelines are required to load the data in the right format. Anecdotally, I've heard that ClickHouse is easier to deploy from this perspective with similar performance, but would love to get others views / experi…
I agree the same. Clickhouse is the best if you are looking for Analytics database.
Re: Druid: A Real-Time Analytical Data Store
#4My sense from the few projects I've seen attempt to use Druid is that there is quite a lot of infrastructure overhead / DevOps support required to manage a cluster at scale, and that fairly complex ingestion pipelines are required to load the data in the right format. Anecdotally, I've heard that ClickHouse is easier to deploy from this perspective with similar performance, but would love to get others views / experi…
I run a ClickHouse cluster. I’ve heard Druid is more difficult, but ClickHouse isn’t exactly a piece of cake. It’s great for a while, but sometimes you can get bitten by weird states. Still, compared to everything I’ve used, for the use case where it excels, it really excels.
Re: Druid: A Real-Time Analytical Data Store
#5database track
Re: Druid: A Real-Time Analytical Data Store
#6My sense from the few projects I've seen attempt to use Druid is that there is quite a lot of infrastructure overhead / DevOps support required to manage a cluster at scale, and that fairly complex ingestion pipelines are required to load the data in the right format. Anecdotally, I've heard that ClickHouse is easier to deploy from this perspective with similar performance, but would love to get others views / experi…
This is one of the reasons why we chose ClickHouse as our backend at PostHog. We wanted something that was relatively simple to operate for our users who deploy PostHog on-prem. We've been super happy with it so far. Still not turn key in many ways, but it's been pretty great.
Re: Druid: A Real-Time Analytical Data Store
#7My sense from the few projects I've seen attempt to use Druid is that there is quite a lot of infrastructure overhead / DevOps support required to manage a cluster at scale, and that fairly complex ingestion pipelines are required to load the data in the right format. Anecdotally, I've heard that ClickHouse is easier to deploy from this perspective with similar performance, but would love to get others views / experi…
Agreed- we use Druid, it's quite successful, but the Druid hosting situation up until this point has been pretty lousy.
Re: Druid: A Real-Time Analytical Data Store
#8My sense from the few projects I've seen attempt to use Druid is that there is quite a lot of infrastructure overhead / DevOps support required to manage a cluster at scale, and that fairly complex ingestion pipelines are required to load the data in the right format. Anecdotally, I've heard that ClickHouse is easier to deploy from this perspective with similar performance, but would love to get others views / experi…
ClickHouse is great. Been using it without any major issues.
We started experiencing some issues with data duplication with ClickHouse when we moved our table to a Sharded+Replicated setup.
Optimize with DEDUPLICATE helped us a lot.. and we can just run this on a Partition instead of the full table.
https://clickhouse.com/docs/en/sql-reference/statements/opti...
ClickHouse is a very powerful system but it's not just setup and forget type.
Re: Druid: A Real-Time Analytical Data Store
#9My sense from the few projects I've seen attempt to use Druid is that there is quite a lot of infrastructure overhead / DevOps support required to manage a cluster at scale, and that fairly complex ingestion pipelines are required to load the data in the right format. Anecdotally, I've heard that ClickHouse is easier to deploy from this perspective with similar performance, but would love to get others views / experi…
Strange - we use Druid at scale at work, and it doesn't really require much maintenance. In fact, for the most part aside from some incidents caused by our own misconfiguration, it buzzes away just fine. Runs on Kubernetes. Most annoying thing was tuning segment compaction.
It's actually quite nice - because since everything is stored on GCS/S3, it is mostly self healing, we can treat the historical as cattle and not pets.
We also run clickhouse, and unfortunately the above is not true - at least in our setup.
Re: Druid: A Real-Time Analytical Data Store
#10My sense from the few projects I've seen attempt to use Druid is that there is quite a lot of infrastructure overhead / DevOps support required to manage a cluster at scale, and that fairly complex ingestion pipelines are required to load the data in the right format. Anecdotally, I've heard that ClickHouse is easier to deploy from this perspective with similar performance, but would love to get others views / experi…
Druid is horizontally scalable by itself if you have access to something like S3 or any compatible Object Storage. Druid's core design is remarkable, designed from the ground up to optimally leverage and work harmoniously with cloud tech. Once it is setup appropriately for your use-case, it's trivial to stamp out over and over with Terraform.