Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ClickHouse gets lazier and faster: Introducing lazy materialization

clickhouse.com

Post reply on HN
Print view View original

91–100 of 130 posts

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#91

Post by nasretdinov »

theLiminator wrote: post #41
jurgenkesker wrote: post #21

I really like Clickhouse. Discovered it recently, and man, it's such a breath of fresh air compared to suboptimal solutions I used for analytics. It's so fast and the CLI is also a joy to work with.

How does it compare to duckdb and/or polars?

In my understanding DuckDB doesn't have its own optimised storage that can accept writes (in a sense that ClickHouse does, where it's native storage format gives you best performance), and instead relies on e.g. reading data from Parquet and other formats. That makes sense for an embedded analytics engine on top of existing files, but might be a problem if you wanted to use DuckDB e.g. for real-time analytics where the inserted data needs to be available for querying in a few seconds after it's been inserted. ClickHouse was designed for the latter use case, but at a cost of being a full-fledged standalone service by design. There are embedded versions of ClickHouse, but they are much bulkier and generally less ergonomic to use (although that's a personal preference)

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#93

Post by nasretdinov »

rfoo wrote: post #34
baq wrote: post #13

Earlier quoted context omitted.

Slow VMs on overprovisioned cloud hosts which cost as much per month as a dedicated box per year have broken a generation of engineers. You could host so much from your macbook. The average HN startup could be hosted on a $200 minipc from a closet for the first couple of years if not more - and I'm talking expensive here for the extra RAM you want to not restart every hour when you have a memory leak.

> so much from your macbook At least on cloud I can actually have hundreds of GiBs of RAM. If I want this on my Macbook it's even more expensive than my cloud bill.

Strangely I've found inverse to be true: many backend technologies are actually quite good with memory management and often require as little as a few GiB of RAM or even less to serve production traffic. Often a single IDE consumes more RAM than a production Go binary that serves thousands of requests per second for example

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#94

Post by nasretdinov »

skeptrune wrote: post #65
mmsimanga wrote: post #45

IMHO if ClickHouse had Windows native release that does not need WSL or a Linux virtual machine it would be more popular than DuckDB. I remember for years MySQL being way more popular than PostgreSQL. One of the reasons being MySQL had a Windows installer.

Is Clickhouse not already more popular than DuckDB?

28k stars on GitHub for DuckDB vs 40k for ClickHouse - pretty close. But, anecdotally, here on HN DuckDB gets mentioned much more often

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#95

Post by nasretdinov »

meta_ai_x wrote: post #30

can we take the "packing your luggage" analogy and only pack the things we actually use in the trip and apply that to clickhouse?

Are you implying that ClickHouse is too large? You can build ClickHouse with most features disabled, it must be much smaller if you do that.

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#96

Post by olavgg »

rastignack wrote: post #81
memset wrote: post #64

Earlier quoted context omitted.

chdb and clickhouse-local is nearly as ergonomic as duckdb with all of the features of ch. duckdb has unfortunately been leaning away from pure oss - the ui they released is entirely hosted on motherduck’s servers (which, while an awesome project, makes me feel like the project will be cannibalized by a proprietary extensions.)

ClickHouse too. Their sharedmergetree is not open source at all. It makes ClickHouse OSS obsolete design-wise. Shame.

Actually this is not important. Because they need to make a living, and with that they have to make some functonality of ClickHouse propritary.

You can scale the pure open source project really far. And if you need more, you do have money to pay for it.

People need to stop thinking that open source is free leech. Open source is about sharing knowledge and building trust between each other. But we are still living in a business world where we are competing.

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#97

Post by mmsimanga »

codedokode wrote: post #84
mmsimanga wrote: post #45

IMHO if ClickHouse had Windows native release that does not need WSL or a Linux virtual machine it would be more popular than DuckDB. I remember for years MySQL being way more popular than PostgreSQL. One of the reasons being MySQL had a Windows installer.

I was under impression that servers and databases generally run on Linux though.

Windows still runs on 71% of the desktop and laptops [1]. In my experience a good number of applications start life on simple desktops and then graduate to servers if they are successful. I work in the field of analytics. I have a locked down Windows desktop and I have been able to try out all the other databases such as MySQL, MariaDB, PostgreSQL and DuckDB because they have windows installers or portable apps. I haven't been able to try out ClickHouse. This is my experience and YMMV.

[1]https://en.wikipedia.org/wiki/Usage_share_of_operating_syste...

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#98

Post by rastignack »

olavgg wrote: post #96
rastignack wrote: post #81

Earlier quoted context omitted.

ClickHouse too. Their sharedmergetree is not open source at all. It makes ClickHouse OSS obsolete design-wise. Shame.

Actually this is not important. Because they need to make a living, and with that they have to make some functonality of ClickHouse propritary. You can scale the pure open source project really far. And if you need more, you do have money to pay for it. People need to stop thinking that open source is free leech. Open source is about sharing knowledge and building trust between each other. But we are still living in…

It is important to decorelate compute and Storage.

I think people should be aware that the Core Storage work is not beeing done in the open anymore. The process of letting the open source core grow useless while selling proprietary addons is fine. I just have a problem with people calling this open source.

Some prefer using open source. Are they leeches ?

Regarding scaling open source projects, did Linux scale ?

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#99

Post by tnolet »

We adopted ClickHouse ~4 years ago. We COULD have stayed on just Postgres. With a lot of bells, whistles, aggregation, denormalisation, aggressive retention limits and job queues etc. we could have gotten acceptable response times for our interactive dashboard.

But we chose ClickHouse and now we just pump in data with little to no optimization.

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#100

Post by nasretdinov »

tnolet wrote: post #99

We adopted ClickHouse ~4 years ago. We COULD have stayed on just Postgres. With a lot of bells, whistles, aggregation, denormalisation, aggressive retention limits and job queues etc. we could have gotten acceptable response times for our interactive dashboard. But we chose ClickHouse and now we just pump in data with little to no optimization.

I imagine with Postgres there's also an option of using a plugin like Greenplum or something else, which may help to bridge the gap, but probably not to the level of ClickHouse.
Post reply on HN