11–20 of 29 posts
We recently added a MySQL/MariaDB CDC connector in ClickPipes on ClickHouse Cloud. This would have simplified your migration from MariaDB.
https://clickhouse.com/docs/integrations/clickpipes/mysql https://clickhouse.com/docs/integrations/clickpipes/mysql/so...
How does Clickhouse compare to Druid, Pinot or Star Tree?
https://www.onehouse.ai/blog/apache-spark-vs-clickhouse-vs-p...
Druid is real-time analytics, similar to Clickhouse. StarRocks is best at Joins - Clickhouse is not good for joins.
How does Clickhouse compare to Druid, Pinot or Star Tree?
Here's a good performance study by OneHouse comparing Clickhouse, StarRocks, Trino: https://www.onehouse.ai/blog/apache-spark-vs-clickhouse-vs-p... Druid is real-time analytics, similar to Clickhouse. StarRocks is best at Joins - Clickhouse is not good for joins.
This is less and less true as time goes on tbh. 25.9 introduced Join Reordering as well - https://clickhouse.com/blog/clickhouse-release-25-09
Seems weird not to use Redis as the buffering layer + minutely cron job. Seems a lot simpler than installing Kafka + Vector.
Kafka and Redis is a "pick your poison" IMO, scaling and operating those have their own headaches.
BTW you could've used e.g. kittenhouse ( https://github.com/YuriyNasretdinov/kittenhouse , my fork) or just a simpler buffer table, with 2 layers and a larger aggregation period than in the example. Alternatively, you could've used async insert functionality built into ClickHouse: https://clickhouse.com/docs/optimize/asynchronous-inserts . All of these solutions are operationally simpler than Kafka + Vector, although…
There were a lot of simpler options that came to mind while reading through this, frankly. But I imagine the writeup eschews myriad future concerns and does not entirely illustrate the pressure and stress of trying to solve such a high-scale problem. Ultimately, going with a somewhat more complex solution that involves additional architecture but has been tried and tested by a 3rd party that you trust can sometimes b…
1a) If you’re still having too many files/parts, then fix your partition by, and mergetree primary key.
2) why are you writing to kafka when vector dev does buffering / batching?
3) if you insist on kafka, https://clickhouse.com/docs/engines/table-engines/integratio... consumes directly from kafka (or since you’re on CHC, use clickhouse pipes) — what’s the point of vector here?
Your current solution is unnecessarily complex. I’m guessing the core problem is your merge tree primary key is wrong.
1) clickhouse async_insert would have solved all your issues: https://clickhouse.com/docs/optimize/asynchronous-inserts 1a) If you’re still having too many files/parts, then fix your partition by, and mergetree primary key. 2) why are you writing to kafka when vector dev does buffering / batching? 3) if you insist on kafka, https://clickhouse.com/docs/engines/table-engines/integratio... consumes directly from kafka (…
From experience the Kafka tables in ClickHouse are not stable at a high volumes, and harder to debug when things go sideways. It is also easier to mutate your data before ingestion using Vector's VRL scripting language vs. ClickHouse table views (SQL) when dealing with complex data that needs to be denormalized into a flat table.
1) clickhouse async_insert would have solved all your issues: https://clickhouse.com/docs/optimize/asynchronous-inserts 1a) If you’re still having too many files/parts, then fix your partition by, and mergetree primary key. 2) why are you writing to kafka when vector dev does buffering / batching? 3) if you insist on kafka, https://clickhouse.com/docs/engines/table-engines/integratio... consumes directly from kafka (…
Writing to Kafka allowed them to continue their current ingestion process into MariaDB at the same time as ClickHouse. Kafka consumer groups allow the data to be consumed twice by different consumer pools that have different throughput without introducing bottlenecks. From experience the Kafka tables in ClickHouse are not stable at a high volumes, and harder to debug when things go sideways. It is also easier to muta…
The one they're going to shut down as soon as this works? Yeah, great reason to make a permanent tech choice for a temporary need. Versus just keeping the MariaDB stuff exactly the same on the PHP side and writing to 2 destinations until cutover is achieved. Kafka is wholly unnecessary here. Vector is great tech but likely not needed. Kafka + Vector is absolutely the incorrect solution.
Their core problem is the destination table schema (which they did not provide) and a very poorly chosen primary key + partition.
I setup some Vector to buffer ElasticSearch writes years ago, also for logs, it ran so well without any problems that I almost fogot about it.
can you just buffer some writes in Vector and eliminate Kafka? I setup some Vector to buffer ElasticSearch writes years ago, also for logs, it ran so well without any problems that I almost fogot about it.