Live data from Hacker News

Scaling product analytics built on ClickHouse

posthog.com

11–13 of 13 posts

Re: Scaling product analytics built on ClickHouse

#11
post #8
post #4

ClickHouse is awesome, but most of the benefits come from columnar storage and you need to design around that. Be aware of how the thing works and how computer architecture works, because sympathy with the machine is what reaps rewards. You want to minimize the number and size of columns touched when filtering and aggregating. If you need source data, store it relationally or in a document store and only select the k…

We store billions of JSON in CH and it performs beautifully. Not sure where you’re getting your experience from.

If you look up a single blob by id, you're going to have a bad time.

My experience comes from using CH as an secondary store for relational data where it could be filtered, sorted and aggregated much faster. The downside was that pulling out all the columns resulted in poor performance - like I said, it was just as slow as a relational store when doing a SELECT *.

Re: Scaling product analytics built on ClickHouse

#12
post #6
post #4

ClickHouse is awesome, but most of the benefits come from columnar storage and you need to design around that. Be aware of how the thing works and how computer architecture works, because sympathy with the machine is what reaps rewards. You want to minimize the number and size of columns touched when filtering and aggregating. If you need source data, store it relationally or in a document store and only select the k…

Perhaps your experience is different from mine but JSON string columns are very common in ClickHouse. A common schema idiom is to put the JSON in a string and then selectively materialize commonly queried properties into regular table columns. You can index the column with a bloom filter index to do needle-in-a-haystack searches on the entire row's data. Bloom filters are tricky to parameterize correctly but I have s…

Yeah, I think my exploration of the design space is different to other people who are using CH to e.g. store event data.

If you can eliminate most "rows" with criteria on materialized columns, then the occasional dip into JSON shouldn't be too bad.

Re: Scaling product analytics built on ClickHouse

#13
post #11
post #8

Earlier quoted context omitted.

We store billions of JSON in CH and it performs beautifully. Not sure where you’re getting your experience from.

If you look up a single blob by id, you're going to have a bad time. My experience comes from using CH as an secondary store for relational data where it could be filtered, sorted and aggregated much faster. The downside was that pulling out all the columns resulted in poor performance - like I said, it was just as slow as a relational store when doing a SELECT *.

This is impossible. I suggest reading through CH's json functions and seeing how they combine with materialized views.

ex: https://eng.uber.com/logging/

Post reply on HN