61–70 of 100 posts
Earlier quoted context omitted.
What do you do to then query the data? I usually need indexes so queries are not slow. Perhaps I could insert into a staging table then bulk copy the data over to an indexed table, but that seems silly.
Could replicating to a DB with indexing (purely for queries) work?
You could use partitions though.
I feel like with all the Clickhouse praise on HN that we /must/ be doing something fundamentally wrong because I hate every interaction I have with Clickhouse. * Timeouts (only 30s???) unless I used the cli client * Cancelling rows - Just kill me, so many bugs and FINAL/PREWHERE are massive foot-guns * Cluster just feels annoying and fragile don't forget "ON CLUSTER" or you'll have a bad time Again, I feel like we mu…
From the docs on FINAL: > However, using FINAL is sometimes necessary in order to produce accurate results Welp.
It’s a table design optimised for specific workloads, and the docs and design detail those tradeoffs.
We use it at work for workloads that can tolerate “retreading” over stale data, because it means they can efficiently write to the db without round tripping, or locking and row updates, and without the table growing massive. It works fantastically in our use case.
Earlier quoted context omitted.
Interesting about "Timeouts (only 30s???)" - most likely, this is a limitation configured explicitly for a user on your server. You can set it up with the `max_execution_time`, and by default, it is unlimited. For example, I've set it up, along with many more limitations for my public playground https://play.clickhouse.com/ , and it allows me to, at least, make it public and not worry much. It could also be a configu…
I can believe it's a config issue, I'll have to look into it. I didn't setup the cluster/dbs and when I asked about I was told "use the cli". I'll try to see if I can get that fixed.
How were they doing millions of events per minute with postgres. I'm struggling with pg write performance ATM and want some tips.
I have a tangentially related question since I don’t use an Olap db: is deleting data so hard to perform? Is it necessarily an immutable storage? If so, is it a gdpr compliant storage solution? I am asking it since gdpr compliance may require data deletion (or at least anonimization)
This in turn, creates read+write load. Modern OLAP db’s often support it, often via mitigating strategies to minimise the amount of extra work they incur: mark tainted rows, exclude them from queries, and clean up asynchronously; etc.
Earlier quoted context omitted.
It's not actually so esoteric. The two main knobs are - max_concurrent_queries, since each query uses a certain amount of memory - max_memory_usage, which is the max per-query memory usage Here's my full config for running clickhouse on a 2GiB server without OOMs. Some stuff in here is likely irrelevant, but it's a starting point. diff --git a/clickhouse-config.xml b/clickhouse-config.xml index f8213b65..7d7459cb 100…
> The two main knobs are my experience is that those are not enough, multiple algorithms will just fail saying you hit max memory limit. There are many other knobs, for example: when to start external aggregation or sorting. For some cases I couldn't figure out setup and query just hits OOM without any ideas how to fix it.
What kind of queries are you trying to do? Also, what kind of machine are you running on?
Earlier quoted context omitted.
> The two main knobs are my experience is that those are not enough, multiple algorithms will just fail saying you hit max memory limit. There are many other knobs, for example: when to start external aggregation or sorting. For some cases I couldn't figure out setup and query just hits OOM without any ideas how to fix it.
How is your table setup? It’s plausible the on-disk/index layout is not amenable to the kinds of queries you’re trying to do. What kind of queries are you trying to do? Also, what kind of machine are you running on?
> Recently, the most interesting rift in the Postgres vs OLAP space is [Hydra]( https://www.hydra.so ), an open-source, column-oriented distribution of Postgres that was very recently launched (after our migration to ClickHouse). Had Hydra been available during our decision-making time period, we might’ve made a different choice. There will likely be a good OLAP solution (possibly implemented as an extension) in Post…
> Recently, the most interesting rift in the Postgres vs OLAP space is [Hydra]( https://www.hydra.so ), an open-source, column-oriented distribution of Postgres that was very recently launched (after our migration to ClickHouse). Had Hydra been available during our decision-making time period, we might’ve made a different choice. There will likely be a good OLAP solution (possibly implemented as an extension) in Post…
Have you seen: https://benchmark.clickhouse.com/