Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ClickHouse gets lazier and faster: Introducing lazy materialization

clickhouse.com

Post reply on HN
Print view View original

61–70 of 130 posts

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#61

Post by ksec »

pests wrote: post #58
ksec wrote: post #55

Earlier quoted context omitted.

Lots of Google analytics competitors appeared between 2017 and 2023 due to privacy reasons. And a lot of them started with normal Postgres or MySQL then switched to Clickhouse or simply started with Clickhouse knowing they could scale far better. At least in terms of capability and reputation it was already well known by 2021 and certainly not legacy or bulky. At least on HN clickhouse is very often submitted and rea…

Perhaps not legacy/bulky buy it maybe... enterprisey? I just remember having the same reaction to CH as to hearing Oracle.

Or may be Heavy duty? Although I remember a lot of people were sceptical of CH simply because it came from Yandex from Russia. And that was before the war.

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#62

Post by skeptrune »

simonw wrote: post #2

Unrelated to the new materialization option, this caught my eye: "this query sorts all 150 million values in the helpful_votes column (which isn’t part of the table’s sort key) and returns the top 3, in just 70 milliseconds cold (with the OS filesystem cache cleared beforehand) and a processing throughput of 2.15 billion rows/s" I clearly need to update my mental model of what might be a slow query against modern har…

Strong and up to date intuition on "slow vs. fast" queries is an underrated software engineering skill. Reading blogs like this one is worth it just for that alone.

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#63

Post by auxten »

AYBABTME wrote: post #52
simonw wrote: post #51

Earlier quoted context omitted.

They have an interesting version that's packaged a bit like DuckDB - you can even "pip install" it: https://github.com/chdb-io/chdb

They don't do static builds AFAICT, which would make it a real competitor to DuckDB.

chDB author here, You are right, we have not made a static libchDB. BTW, I guess you are a golang developer?

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#64

Post by memset »

dangoodmanUT wrote: post #46

God clickhouse is such great software, if it only it was as ergonomic as duckdb, and management wasn't doing some questionable things (deleting references to competitors in GH issues, weird legal letters, etc.) The CH contributors are really stellar, from multiple companies (Altinity, Tinybird, Cloudflare, ClickHouse)

chdb and clickhouse-local is nearly as ergonomic as duckdb with all of the features of ch.

duckdb has unfortunately been leaning away from pure oss - the ui they released is entirely hosted on motherduck’s servers (which, while an awesome project, makes me feel like the project will be cannibalized by a proprietary extensions.)

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#65

Post by skeptrune »

mmsimanga wrote: post #45

IMHO if ClickHouse had Windows native release that does not need WSL or a Linux virtual machine it would be more popular than DuckDB. I remember for years MySQL being way more popular than PostgreSQL. One of the reasons being MySQL had a Windows installer.

Is Clickhouse not already more popular than DuckDB?

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#66

Post by skeptrune »

>Despite the airport drama, I’m still set on that beach holiday, and that means loading my eReader with only the best.

What a nice touch. Technical information and diagrams in this were top notch, but the fact there was also some kind of narrative threaded in really put it over the top for me.

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#67

Post by FridgeSeal »

ramraj07 wrote: post #44
baq wrote: post #13

Earlier quoted context omitted.

Slow VMs on overprovisioned cloud hosts which cost as much per month as a dedicated box per year have broken a generation of engineers. You could host so much from your macbook. The average HN startup could be hosted on a $200 minipc from a closet for the first couple of years if not more - and I'm talking expensive here for the extra RAM you want to not restart every hour when you have a memory leak.

I don't see how that's the root cause. ClickHouse and snowflake run on your so-called slow vms on overprovisioned cloud hosts and they're efficient as hell. It's all about your optimizations. The real problem is the lack of understanding by most engineers the degree of overprovisioning they do for code that's simple and doing stupid things using an inefficient 4th order language on top of 5 different useless (imo) ab…

No but my developer velocity! We should sacrifice literally everything else in order to enable me!!!! Nothing else matters except for my ease of life!

/s

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#68

Post by FridgeSeal »

vjerancrnjak wrote: post #53
apavlo wrote: post #43

Earlier quoted context omitted.

> It's quite amazing how a db like this shows that all of those row-based dbs are doing something wrong They're not "doing something wrong". They are designed differently for different target workloads. Row-based -> OLTP -> "Fetch the entire records from order table where user_id = XYZ" Column-based -> OLAP -> "Compute the total amount of orders from the order table grouped by month/year"

Filtering by user id would also be trivially fast. It’s transactions mostly that make things slow. Like various isolation levels, failures if stale data was read in a transaction etc. I understand the difference, just a shame there’s nothing close to read or write rate , even on an index structure that has a copy of the columns. I’m aware that similar partitioning is available and that improves write and read rate bu…

Some of the “new SQL” hybrid (HTAP, hybrid transaction-analytical processing) databases might be of interest to you. TiDB is the main example off the top of my head.

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#69

Post by shmerl »

jurgenkesker wrote: post #21

I really like Clickhouse. Discovered it recently, and man, it's such a breath of fresh air compared to suboptimal solutions I used for analytics. It's so fast and the CLI is also a joy to work with.

What's the story with the Debian package for it? It was removed as unmaintained.

Re: ClickHouse gets lazier and faster: Introducing lazy materialization

#70

Post by kevinventullo »

amluto wrote: post #4
simonw wrote: post #2

Unrelated to the new materialization option, this caught my eye: "this query sorts all 150 million values in the helpful_votes column (which isn’t part of the table’s sort key) and returns the top 3, in just 70 milliseconds cold (with the OS filesystem cache cleared beforehand) and a processing throughput of 2.15 billion rows/s" I clearly need to update my mental model of what might be a slow query against modern har…

> I guess sorting 150 million integers in 70ms shouldn't be surprising. I find sorting 150M integers at all to be surprising. The query asks for finding the top 3 elements and returning those elements, sorted. This can be done trivially by keeping the best three found so far and scanning the list. This should operate at nearly the speed of memory and use effectively zero additional storage. I don’t know whether Click…

With non-mutable “streaming” input, there is an O(n) algorithm to obtain the unsorted top k with only O(k) extra memory.

The basic idea is to maintain a buffer of size 2k, run mutable unsorted top k on that, drop the smaller half (i.e the lowest k elements), then stream in the next k elements from the main list. Each iteration takes O(k), but you’re processing k elements at a time, so overall runtime is O(n).

When you’re done, you can of course sort for an additional k*log(k) cost.

Post reply on HN