Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ClickHouse gets lazier and faster: Introducing lazy materialization

clickhouse.com

Post reply on HN
Print view View original

121–130 of 130 posts

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#121

Post by codedokode »

mmsimanga wrote: post #97
codedokode wrote: post #84

Earlier quoted context omitted.

I was under impression that servers and databases generally run on Linux though.

Windows still runs on 71% of the desktop and laptops [1]. In my experience a good number of applications start life on simple desktops and then graduate to servers if they are successful. I work in the field of analytics. I have a locked down Windows desktop and I have been able to try out all the other databases such as MySQL, MariaDB, PostgreSQL and DuckDB because they have windows installers or portable apps. I ha…

Fair point, but if your desktop is locked down then you might not be able to use administrator privileges that many programs require for installation (especially software that uses DRM and licenses). So you might be not able to run the software even if there was a Windows version.

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#122

Post by theLiminator »

nasretdinov wrote: post #118
theLiminator wrote: post #117

Earlier quoted context omitted.

Nah, duckdb does have their own format (not sure if it's write friendly, though i believe it is).

It does, but the performance isn't great apparently: https://github.com/duckdb/duckdb/discussions/10161

Yeah, I don't really know. Though in the OLAP space that issue/discussion is really old. There's a good chance that performance is dramatically better now though YMMV.

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#124

Post by thenaturalist »

theLiminator wrote: post #117
nasretdinov wrote: post #91

Earlier quoted context omitted.

In my understanding DuckDB doesn't have its own optimised storage that can accept writes (in a sense that ClickHouse does, where it's native storage format gives you best performance), and instead relies on e.g. reading data from Parquet and other formats. That makes sense for an embedded analytics engine on top of existing files, but might be a problem if you wanted to use DuckDB e.g. for real-time analytics where t…

Nah, duckdb does have their own format (not sure if it's write friendly, though i believe it is).

It cannot do concurrent writes natively and that is not a design goal of the DuckDB creators. See: https://duckdb.org/docs/stable/connect/concurrency.html#writ...

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#125

Post by spatular »

theLiminator wrote: post #41
jurgenkesker wrote: post #21

I really like Clickhouse. Discovered it recently, and man, it's such a breath of fresh air compared to suboptimal solutions I used for analytics. It's so fast and the CLI is also a joy to work with.

How does it compare to duckdb and/or polars?

Clickhouse is a network server, duckdb and polars are in-process databases. It's like postgres vs sqllite.

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#126

Post by riku_iki »

fermuch wrote: post #116
xiasongh wrote: post #88

Has anyone compared ClickHouse and StarRocks[0]? Join performance seems a lot better on StarRocks a few months ago but I'm not sure if that still holds true. [0] https://www.starrocks.io/

Yes! There is a benchmark on ClickBench: https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQWxsb3lEQi...

But clickbench doesn't have joins..

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#128

Post by valyala »

codedokode wrote: post #78
ww520 wrote: post #74

Earlier quoted context omitted.

That’s true. With such a small data domain, there would be a lot repeated numbers in the 160M values, leading to highly compressible data.

I found in the article that the column uses 70 Mb of storage. if it was sorted (i.e. if it was an index) it would take even much less space. I don't understand though how they loaded 70 Mb of data with 125 MiB/s SSD in 70 ms.

There is no need in loading data block, which has no rows with column values, which might be included into the final set of rows. If every column in every granule has a header containing the minimum and the maximum value seen in the granule, then ClickHouse can read and check only the column header per every granule, without the need to read the column data.

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#129

Post by theLiminator »

spatular wrote: post #125
theLiminator wrote: post #41

Earlier quoted context omitted.

How does it compare to duckdb and/or polars?

Clickhouse is a network server, duckdb and polars are in-process databases. It's like postgres vs sqllite.

There's chdb though.

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#130

Post by AlexClickHouse »

ignoreusernames wrote: post #109
kwillets wrote: post #77

Late Materialization, 19 years later. https://dspace.mit.edu/bitstream/handle/1721.1/34929/MIT-CSA...

Same thing with columnar/vectorized execution. It has been known for a long time that's the "correct" way to process data for olap workflows, but only became "mainstream" in the last few years(mostly due to arrow). It's awesome that clickhouse is adopting it now, but a shame that it's not standard on anything that does analytics processing.

ClickHouse predates Apache Arrow.
Post reply on HN