Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ClickHouse gets lazier and faster: Introducing lazy materialization

clickhouse.com

Post reply on HN
Print view View original

111–120 of 130 posts

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#111

Post by tnolet »

nasretdinov wrote: post #100
tnolet wrote: post #99

We adopted ClickHouse ~4 years ago. We COULD have stayed on just Postgres. With a lot of bells, whistles, aggregation, denormalisation, aggressive retention limits and job queues etc. we could have gotten acceptable response times for our interactive dashboard. But we chose ClickHouse and now we just pump in data with little to no optimization.

I imagine with Postgres there's also an option of using a plugin like Greenplum or something else, which may help to bridge the gap, but probably not to the level of ClickHouse.

yes, we looked at Timescale also. But they were much younger then and Clickhouse was more mature. Clickhouse cloud did not exist yet. We now use a mix of Altinity and Cloud.

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#112

Post by kwillets »

ignoreusernames wrote: post #109
kwillets wrote: post #77

Late Materialization, 19 years later. https://dspace.mit.edu/bitstream/handle/1721.1/34929/MIT-CSA...

Same thing with columnar/vectorized execution. It has been known for a long time that's the "correct" way to process data for olap workflows, but only became "mainstream" in the last few years(mostly due to arrow). It's awesome that clickhouse is adopting it now, but a shame that it's not standard on anything that does analytics processing.

Nothing in C-store seems to have sunk in. In clickhouse's case I can forgive them since it was an open source, bootstrap type of project, and their cash infusion seems to be going into basic re-engineering, but in general slowly re-implementing Vertica seems like a flawed business model.

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#113

Post by [deleted] »

dangoodmanUT wrote: post #46

God clickhouse is such great software, if it only it was as ergonomic as duckdb, and management wasn't doing some questionable things (deleting references to competitors in GH issues, weird legal letters, etc.) The CH contributors are really stellar, from multiple companies (Altinity, Tinybird, Cloudflare, ClickHouse)

[deleted]

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#114

Post by [deleted] »

tnolet wrote: post #99

We adopted ClickHouse ~4 years ago. We COULD have stayed on just Postgres. With a lot of bells, whistles, aggregation, denormalisation, aggressive retention limits and job queues etc. we could have gotten acceptable response times for our interactive dashboard. But we chose ClickHouse and now we just pump in data with little to no optimization.

[deleted]

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#116

Post by fermuch »

xiasongh wrote: post #88

Has anyone compared ClickHouse and StarRocks[0]? Join performance seems a lot better on StarRocks a few months ago but I'm not sure if that still holds true. [0] https://www.starrocks.io/

Yes! There is a benchmark on ClickBench: https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQWxsb3lEQi...

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#117

Post by theLiminator »

nasretdinov wrote: post #91
theLiminator wrote: post #41

Earlier quoted context omitted.

How does it compare to duckdb and/or polars?

In my understanding DuckDB doesn't have its own optimised storage that can accept writes (in a sense that ClickHouse does, where it's native storage format gives you best performance), and instead relies on e.g. reading data from Parquet and other formats. That makes sense for an embedded analytics engine on top of existing files, but might be a problem if you wanted to use DuckDB e.g. for real-time analytics where t…

Nah, duckdb does have their own format (not sure if it's write friendly, though i believe it is).

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#118

Post by nasretdinov »

theLiminator wrote: post #117
nasretdinov wrote: post #91

Earlier quoted context omitted.

In my understanding DuckDB doesn't have its own optimised storage that can accept writes (in a sense that ClickHouse does, where it's native storage format gives you best performance), and instead relies on e.g. reading data from Parquet and other formats. That makes sense for an embedded analytics engine on top of existing files, but might be a problem if you wanted to use DuckDB e.g. for real-time analytics where t…

Nah, duckdb does have their own format (not sure if it's write friendly, though i believe it is).

It does, but the performance isn't great apparently: https://github.com/duckdb/duckdb/discussions/10161

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#119

Post by eru »

porridgeraisin wrote: post #75
senderista wrote: post #37

Earlier quoted context omitted.

How do you efficiently track the "worst element" without something like a max-heap? But yeah, this is a fun algorithm. I think I've seen it before but can't place it, do you remember where you came across it?

if x > worst then worst = x

Exactly. Keeping a max (or min or sum etc) is easy, if you only ever insert into your structure.

The deletion comes in a big batch, where we don't mind paying a linear cost to prune and rebuild our bucket.

Oh, and in our case it's even simpler: the worst element in our buffer only updates during the pruning phase. By construction, we otherwise only ever insert elements that are better than the worst, so we don't have to update the worst. (Outside of the first k elements that we all take anyway to get started. But if you want, you can handle that as a special case.)

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#120

Post by senderista »

eru wrote: post #119
porridgeraisin wrote: post #75

Earlier quoted context omitted.

if x > worst then worst = x

Exactly. Keeping a max (or min or sum etc) is easy, if you only ever insert into your structure. The deletion comes in a big batch, where we don't mind paying a linear cost to prune and rebuild our bucket. Oh, and in our case it's even simpler: the worst element in our buffer only updates during the pruning phase. By construction, we otherwise only ever insert elements that are better than the worst, so we don't have…

Oh duh, you only evict when pruning the bottom half.
Post reply on HN