31–40 of 100 posts
I'm curious: how many rows Lago store in its CH cluster? Do they collect data for fighting fraud? PG can handle a billion rows easily.
I super love PG but PG is too far away from that.
I feel like with all the Clickhouse praise on HN that we /must/ be doing something fundamentally wrong because I hate every interaction I have with Clickhouse. * Timeouts (only 30s???) unless I used the cli client * Cancelling rows - Just kill me, so many bugs and FINAL/PREWHERE are massive foot-guns * Cluster just feels annoying and fragile don't forget "ON CLUSTER" or you'll have a bad time Again, I feel like we mu…
What foot guns have you run into with FINAL?
Earlier quoted context omitted.
A couple million (<10M), I don't have a better number available right now. Not all (or even most) of those need cancelling rows thankfully.
You don’t need a cluster nor should you be having any issues you mentioned. I run 10x that volume daily on a single gcp box (8 core / 64GB). We migrated off BigQuery and went from $10k/mo to about $250/mo. And it’s faster for both low-latency and big slow queries.
Earlier quoted context omitted.
What is your use case? If you're deleting rows that already feels like maybe it's not the intended use case. I think about clickhouse as taking in a firehose of immutable data that you want to aggregate/analyze/report on. Let's say a million records per second. I'll make up an example, the orientation, speed and acceleration of every Tesla vehicle in the world in real time every second.
It's to power all our analytics. We ETL data into it and some data is write-once so we don't have updates/deletes but a number of our tables have summary data ETL'd into them which means cleaning up the old rows. I'm sure CH shines for insert-only workloads but that doesn't cover all our needs.
There are many applications that require extremely high insertion rates (millions of records per second), very large total number of rows (billions, trillions) and flexible/fast querying/aggregation with high read rates (100's of millions or higher rows/s) and that's sort of the sweet spot IMO for ClickHouse and where you'll be pressed to find alternatives. I'm sure it can be used in other situations but maybe there are more choices if you're in those.
Earlier quoted context omitted.
You don’t need a cluster nor should you be having any issues you mentioned. I run 10x that volume daily on a single gcp box (8 core / 64GB). We migrated off BigQuery and went from $10k/mo to about $250/mo. And it’s faster for both low-latency and big slow queries.
The plan is to 10x that volume in the not too distant future but given what you've said I can believe we are horribly over-provisioned/over-scaled. Thank you!
Overall I’d say CH isn’t as tolerant or forgiving as BigQuery, Snowflake, or Databricks. You can write the worst SQL possible and BQ will happily charge you $5/TB for that cartesian self-join. CH meanwhile will error with memory limit or even crash.
Earlier quoted context omitted.
The plan is to 10x that volume in the not too distant future but given what you've said I can believe we are horribly over-provisioned/over-scaled. Thank you!
It sounds like you’re probably using Clickhouse Cloud? If so, I was not impressed. Overly pushy sales people, pricing isn’t competitive, and they’re trying to cater to the snowflake/databricks crowd without smoothing any rough edges (like the default timeout being enabled on a GUI). Overall I’d say CH isn’t as tolerant or forgiving as BigQuery, Snowflake, or Databricks. You can write the worst SQL possible and BQ wil…
And if you use MariaDB, just enable columnstore. Why not treat yourself to s3 backed storage while you are there? It is extremely cost effective when you can scale a different workload without migrating.
This is no shade to postgres or maria, but they don’t hold a candle to the simplicity, speed, and cost efficiency of clickhouse for olap needs.
For a lot of what people may want to do, they'd probably notice very little difference between the three.
If your analysis in Postgres was based on multiple tables and required a lot of JOIN operations, I don't think ClickHouse is a good choice. In such cases, you often need to denormalize multiple data tables into one large table in advance, which means complex ETL and maintenance costs.
For these more common scenarios, I think StarRocks (www.StarRocks.io) is a better choice. It's a Linux Foundation open-source project, with single-table query speeds comparable to ClickHouse (you can check Clickbench), and unmatched multi-table join query speeds, plus it can directly query open data lakes.
Earlier quoted context omitted.
It sounds like you’re probably using Clickhouse Cloud? If so, I was not impressed. Overly pushy sales people, pricing isn’t competitive, and they’re trying to cater to the snowflake/databricks crowd without smoothing any rough edges (like the default timeout being enabled on a GUI). Overall I’d say CH isn’t as tolerant or forgiving as BigQuery, Snowflake, or Databricks. You can write the worst SQL possible and BQ wil…
We are using Altinity. I believe it's a 3-server cluster and we have 2 clusters (our prod one and another one we are trying to promote to production once our data integrity checks pass, at which point we will spin the other down).
Is ClickHouse a suitable engine for analyzing events? Absolutely, as long as you're analyzing a large table, its speed is definitely fast enough. However, you might want to consider the cost of maintaining an OSS ClickHouse cluster, especially when you need to scale up, as the operational costs can be quite high. If your analysis in Postgres was based on multiple tables and required a lot of JOIN operations, I don't…
What else could you possibly need? Backing up is built into it with S3 support: https://clickhouse.com/docs/en/operations/backup#configuring...
Upgrades are a breeze: https://clickhouse.com/docs/en/operations/update
People insist that OMG MAINTENANCE I NEED TO PAY THOUSANDS FOR MANAGED is better, when in reality, it is not.