If your data is too big for postgres, it seems like moving straight to Clickhouse is the best option. We have been through an whole array of distributed database technologies, and Clickhouse might be first one that doesn't have too many compromises.
Ten years of ClickHouse in open source
21–30 of 110 posts
Re: Ten years of ClickHouse in open source
#22Clickhouse is *really* gatekeeping the "zero copy replication" where you store data on object-storage and have high availability from the open source version.
How? Have you tried contributing a reasonable implementation with test coverage and it was rejected?
Re: Ten years of ClickHouse in open source
#23ClickHouse recently has been a breath of fresh air compared to using timescaledb for a long time. Although psql is the greatest there is and I really enjoyed the fact that I could rely on a single database system to run everything, when it came to migration maintenance and deployment it's really a pain and it also feels like development on timescaledb is a bit wishy washy with all the structural changes from version…
I was using TimescaleDB some very long time ago, things have changed quite a lot since (it's now even named differently). In my current setup I was thinking on doing both: upgrading postgresql to timescaledb (to archive old data etc.), and to deploy ClickHouse in parallel. I'm still considering whether to go big on PeerDB to get ClickHouse mirror or just deploy it separately without additional fragility layer. Would…
I would just run both and decomission the old one when a) all data is migrated, b) old data is no longer relevant and can be archived
Re: Ten years of ClickHouse in open source
#24Clickhouse is *really* gatekeeping the "zero copy replication" where you store data on object-storage and have high availability from the open source version.
I think that is just the nature of the open core business - but like most such businesses, they're not very clear about how that is what they are, pretending to be open source business instead.
Re: Ten years of ClickHouse in open source
#25The query speed deserves the praise, but the JSON ingestion path has quiet footguns nobody mentions here. Every numeric column comes back as a string over JSONEachRow, so a forgotten Number() cast silently turns arithmetic into string concatenation, and with input_format_skip_unknown_fields enabled a single typo in a column name drops that field with no error at all. Worth wiring an assertion that inserts a row and r…
[dead]
Re: Ten years of ClickHouse in open source
#26The query speed deserves the praise, but the JSON ingestion path has quiet footguns nobody mentions here. Every numeric column comes back as a string over JSONEachRow, so a forgotten Number() cast silently turns arithmetic into string concatenation, and with input_format_skip_unknown_fields enabled a single typo in a column name drops that field with no error at all. Worth wiring an assertion that inserts a row and r…
We’ve done our JSON ingestion by keeping a schema in the app for all the types we expect, and injecting the types into the query builder.
Then as needed we have materialized columns on our different tables.
Re: Ten years of ClickHouse in open source
#27Clickhouse is *really* gatekeeping the "zero copy replication" where you store data on object-storage and have high availability from the open source version.
This is the main driver for their cloud ;-)
Re: Ten years of ClickHouse in open source
#28Re: Ten years of ClickHouse in open source
#29ClickHouse replacing Loki finally made our observability stack feel 'right'. It really is a powerhouse for logs and general analytical queries.
How do you use it for visualization? Do you use ClickStack? or something else?
Still via Grafana. I ran it side-by-side with Loki and despite trying to optimise Loki and using ClickHouse out of the box - it really was shocking how much faster ClickHouse was for every single query (e.g. in the last 12 hours give my the frequency of logs with a particular JSON event or even "find this log entry, then join back and find the number of times a different entry appears within the same correlation_id)
Re: Ten years of ClickHouse in open source
#30Earlier quoted context omitted.
How do you use it for visualization? Do you use ClickStack? or something else?
Still via Grafana. I ran it side-by-side with Loki and despite trying to optimise Loki and using ClickHouse out of the box - it really was shocking how much faster ClickHouse was for every single query (e.g. in the last 12 hours give my the frequency of logs with a particular JSON event or even "find this log entry, then join back and find the number of times a different entry appears within the same correlation_id)
What does the layout in click house look like? Do the input logs need to have a very defined structure?