Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using ClickHouse to scale an events engine

github.com

Post reply on HN
Print view View original

91–100 of 100 posts

Re: Using ClickHouse to scale an events engine

#91

Post by config_yml »

drewda wrote: post #42

This change may make sense for Lago as a hosted multi-tenant service, as offered by Lago the company. Simultaneously this change may not make sense for Lago as an open-source project self-hosted by a single tenant. But that may also mean that it effectively makes sense for Lago as a business... to make it harder to self host. I don't at all fault Lago for making decisions to prioritize their multi-tenant cloud offeri…

Exactly, I've seen this at Sentry where you now have to run Kafka, Clickhouse, Redis, PG, Zookeeper, memcached and what have you. I get it, but the amount of baggage to handle is a bit difficult.

Re: Using ClickHouse to scale an events engine

#93

Post by lysecret »

alooPotato wrote: post #76

We use BigQuery a lot for internal analytics and we've been super happy. I don't see a lot of love for BigQuery on HN and I wonder why. Tons of features, no hassle and easy to throw a bunch of TB at it. I guess maybe the cost?

Yep love it too, especially with external data on GCS. Costs this way are very low. And the convenience is amazing (getting caches you can stream from for every query is a godsend)

Re: Using ClickHouse to scale an events engine

#94

Post by alooPotato »

lysecret wrote: post #93
alooPotato wrote: post #76

We use BigQuery a lot for internal analytics and we've been super happy. I don't see a lot of love for BigQuery on HN and I wonder why. Tons of features, no hassle and easy to throw a bunch of TB at it. I guess maybe the cost?

Yep love it too, especially with external data on GCS. Costs this way are very low. And the convenience is amazing (getting caches you can stream from for every query is a godsend)

What do you mean by the streaming caches?

Re: Using ClickHouse to scale an events engine

#95

Post by mritchie712 »

riku_iki wrote: post #87
snihalani wrote: post #69

Earlier quoted context omitted.

Have you seen: https://benchmark.clickhouse.com/

that benchmark is very weak, they used just 100M rows which is laughable, also no joins have been tested.

no joins is heavily favoring clickhouse (the creator of the benchmark). I'm not sure it's gotten better since I've seriously looked at them, but CH's join performance was really bad.

Re: Using ClickHouse to scale an events engine

#96

Post by hodgesrm »

wiredfool wrote: post #88
breadchris wrote: post #81

ClickHouse is awesome, but as the post shows, some code is involved in getting the data there. I have been working on Scratchdata [1], which makes it easy to try out a column database to optimize aggregation queries (avg, sum, max). We have helped people [2] take their Postgres with 1 billion rows of information (1.5 TB) and significantly reduce their real-time data analysis query time. Because their data was stored…

My first big win for clickhouse was replacing a 1.2tb, billion + row postgresql DB with clickhouse. It was static data with occasional full replacement loads. We got the DB down to ~ 60GB, with query speeds about 45x faster. Now, the postgres schema wasn't ideal, and we could have saved ~ 3x on it with corresponding speed increases for queries with a refactor similar to the clickhouse schema, but that wasn't really e…

This is a nice illustration of the effects of different choices for storage layout and use of compute. ClickHouse blows away single-threaded queries on row-based data for analytic questions. On the other hand PostgreSQL can offer far higher throughput and concurrency when updating a shopping cart.

Re: Using ClickHouse to scale an events engine

#97

Post by philippemnoel »

riku_iki wrote: post #89
philippemnoel wrote: post #71

Earlier quoted context omitted.

ParadeDB founder here. You can see how we compare to other Postgres-based analytical offerings on ClickBench here: https://blog.paradedb.com/pages/introducing_analytics

[flagged]

I did not know that... That's very disingenuous! We'll make sure to keep this in mind as we make our own comparisons. We haven't tuned ParadeDB at all in our published ClickBench benchmarks, so it should be more representative

Re: Using ClickHouse to scale an events engine

#98

Post by philippemnoel »

Mortiffer wrote: post #86
mritchie712 wrote: post #8

> Recently, the most interesting rift in the Postgres vs OLAP space is [Hydra]( https://www.hydra.so ), an open-source, column-oriented distribution of Postgres that was very recently launched (after our migration to ClickHouse). Had Hydra been available during our decision-making time period, we might’ve made a different choice. There will likely be a good OLAP solution (possibly implemented as an extension) in Post…

so Paradedb and Hydra are using same codebase or just similar approach ?

ParadeDB and Hydra are completely different. We're tackling the same problem of bringing analytics inside Postgres, but using different approaches.

ParadeDB integrates industry standards like Arrow, Parquet, DataFusion to offer columnar storage + vectorized processing. Hydra is building on top of Citus Columnar.

You can read about our approach here: https://blog.paradedb.com/pages/introducing_analytics

Re: Using ClickHouse to scale an events engine

#99

Post by wuputah »

riku_iki wrote: post #89
philippemnoel wrote: post #71

Earlier quoted context omitted.

ParadeDB founder here. You can see how we compare to other Postgres-based analytical offerings on ClickBench here: https://blog.paradedb.com/pages/introducing_analytics

[flagged]

That is an incorrect and baseless accusation, we had nothing to do with "Postgres (tuned)". My commits are only in the `hydra` folder. There are no restrictions on how you set up the benchmark in Clickbench and the settings we use there are analogous with what we use on our cloud service for a similar sized instance.

As the linked post points out, the main 'advantage' of the "tuned" benchmark is the indexes, which are tuned specifically to the queries in the benchmark. We do not use indexes in our version of the benchmark, aside from the primary key (which actually provides no performance advantage).

Re: Using ClickHouse to scale an events engine

#100

Post by riku_iki »

wuputah wrote: post #99
riku_iki wrote: post #89

Earlier quoted context omitted.

[flagged]

That is an incorrect and baseless accusation, we had nothing to do with "Postgres (tuned)". My commits are only in the `hydra` folder. There are no restrictions on how you set up the benchmark in Clickbench and the settings we use there are analogous with what we use on our cloud service for a similar sized instance. As the linked post points out, the main 'advantage' of the "tuned" benchmark is the indexes, which ar…

I apologize for falsely claiming that it was Hydra devs who committed Postgres config.

However, I think problem stands: it is your main marketing pitch to compare HydraDB to undertuned PG, it is right on landing page of your project.

> the main 'advantage' of the "tuned" benchmark is the indexes

I am not sure which post you referred to, but unless you or someone else analyzed execution plans for all PG queries in that benchmark and verified that indexes are actually used, it is just speculations without evidence.

Another issue with this comparison is that ClickBench is toy micro-benchmark with just 100M records. Increasing datasize may or may not be beneficial for HydraDB.

Post reply on HN