41–50 of 58 posts
My team is (over|ab)using Elasticsearch and I've had my eye on ClickHouse for a while. However we're going to migrate everything to AWS and I wonder if RedShift could be a good alternative too, since it's now supporting JSON and semi-structured data apparently.
My team is (over|ab)using Elasticsearch and I've had my eye on ClickHouse for a while. However we're going to migrate everything to AWS and I wonder if RedShift could be a good alternative too, since it's now supporting JSON and semi-structured data apparently.
Altinity offers a managed service for ClickHouse on AWS https://altinity.com/cloud-database/
Great software. We are managing terabytes of stocks data and realtime market scanners queries across all market (billions of books and timesales) with hundreds of concurrent requests. We were using kdb before, but clickhouse is more scalable, way cheaper and much more easy to grasp for a newbie.
How much data? And which types of application/apis are connected to CH?
People who've used Clickhouse or other OLAP databases in production & at scale, how do you "interconnect" it with relational data? I'm currently experimenting with Clickhouse, because my dataflow is increasing in size (40M rows right now, doubling every month or so) and my current setup (MongoDB) is at its limits. I would like to migrate the 40M rows to CH, but I also need the metadata for the rows to be in something…
Take a look at query engines like Trino (formerly PrestoSQL) [ https://trino.io/ ]. (Disclaimer: I'm a contributor to Trino). I used it at a previous job to combine data from MongoDB, Kafka, S3 and Postgres to great effect. It tries to push-down as many operations as possible to the source too to improve performance. Full ANSI SQL support over multiple number of backends (Kafka, Cassandra, Postgres, ClickHouse, S3 an…
[1] https://trino.io/blog/2020/12/27/announcing-trino.html
[2] https://prestodb.io/
[3] https://trino.io/Presto_SQL_on_Everything.pdfPeople who've used Clickhouse or other OLAP databases in production & at scale, how do you "interconnect" it with relational data? I'm currently experimenting with Clickhouse, because my dataflow is increasing in size (40M rows right now, doubling every month or so) and my current setup (MongoDB) is at its limits. I would like to migrate the 40M rows to CH, but I also need the metadata for the rows to be in something…
I have not used PostgreSQL myself but the MySQL database engine works great. In some cases queries from ClickHouse to MySQL run faster than they do on MySQL itself. There are other engines as well, e.g., MongoDB.
[1] https://clickhouse.tech/docs/en/engines/table-engines/integr...
[2] https://clickhouse.tech/docs/en/engines/table-engines/integr...
Earlier quoted context omitted.
Altinity offers a managed service for ClickHouse on AWS https://altinity.com/cloud-database/
Thanks but my employer is becoming an AWS partner and we'll contractually need to spend a lot of money, meaning it's better if we use their SaaS offerings directly rather than going through the marketplace or a 3rd party.
Earlier quoted context omitted.
Thanks but my employer is becoming an AWS partner and we'll contractually need to spend a lot of money, meaning it's better if we use their SaaS offerings directly rather than going through the marketplace or a 3rd party.
That's a funny way of making engineering design decisions.
People who've used Clickhouse or other OLAP databases in production & at scale, how do you "interconnect" it with relational data? I'm currently experimenting with Clickhouse, because my dataflow is increasing in size (40M rows right now, doubling every month or so) and my current setup (MongoDB) is at its limits. I would like to migrate the 40M rows to CH, but I also need the metadata for the rows to be in something…
Take a look at query engines like Trino (formerly PrestoSQL) [ https://trino.io/ ]. (Disclaimer: I'm a contributor to Trino). I used it at a previous job to combine data from MongoDB, Kafka, S3 and Postgres to great effect. It tries to push-down as many operations as possible to the source too to improve performance. Full ANSI SQL support over multiple number of backends (Kafka, Cassandra, Postgres, ClickHouse, S3 an…
People who've used Clickhouse or other OLAP databases in production & at scale, how do you "interconnect" it with relational data? I'm currently experimenting with Clickhouse, because my dataflow is increasing in size (40M rows right now, doubling every month or so) and my current setup (MongoDB) is at its limits. I would like to migrate the 40M rows to CH, but I also need the metadata for the rows to be in something…
If you're using it purely for reporting purposes then you really don't want to interconnect it with your OLTP database to support real-time queries. Reason number one is that you don't want some unexpected analytics workload to suddenly impact your production Postgres. Reason number two is that your analytics data model and your OLTP data model are frequently different. Usually your OLAP model needs to know what the value of a given dimension was at the time the event occurred, but if you're linking directly to Postgres then you can only see what the values are right now.
You can also go the other direction if you need the current values of OLTP data joined with your Clickhouse data: schedule an ETL out of Clickhouse and back into your OLTP database and aggregate the data to a reasonable level such that Posgres can handle it without a problem.
What's your actual use case? I'd normally consider a requirement to join OLAP & OLTP data in real time to be a "design smell". I don't mean that there's no value in things like fdw or easier ways to move data around, but you should consider using it to help with the ETL process and not as a real-time interconnect. Keep OLTP & OLAP workloads separate and both of your DBs will be happier.