Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ten years of ClickHouse in open source

clickhouse.com

Post reply on HN
Print view View original

81–90 of 110 posts

Re: Ten years of ClickHouse in open source

#81

Post by murkt »

ksajadi wrote: post #67

For our metrics and autoscaling engine at Cloud 66, we went through 5 iterations before settling on Clickhouse: 1. Redis 2. Cassandra 3. Handrolled: Ruby + RabbitMQ 4. Handrolled: Go + RabbitMQ 5. Clickhouse Every time we reached some limit or huge optimization burdens that were unfeasible. Clickhouse has been rock solid for the past 4 years.

I am having a hard time picturing what could be the problem that you were solving.

Redis, Cassandra, RabbitMQ and Clickhouse. RabbitMQ looks like a black sheep in this lineup.

Re: Ten years of ClickHouse in open source

#82

Post by tills13 »

murkt wrote: post #81
ksajadi wrote: post #67

For our metrics and autoscaling engine at Cloud 66, we went through 5 iterations before settling on Clickhouse: 1. Redis 2. Cassandra 3. Handrolled: Ruby + RabbitMQ 4. Handrolled: Go + RabbitMQ 5. Clickhouse Every time we reached some limit or huge optimization burdens that were unfeasible. Clickhouse has been rock solid for the past 4 years.

I am having a hard time picturing what could be the problem that you were solving. Redis, Cassandra, RabbitMQ and Clickhouse. RabbitMQ looks like a black sheep in this lineup.

Also like redis and clickhouse are diametrically different platforms. Wild to go from one to the other.

Re: Ten years of ClickHouse in open source

#83

Post by tills13 »

jaysh wrote: post #5

ClickHouse replacing Loki finally made our observability stack feel 'right'. It really is a powerhouse for logs and general analytical queries.

We are fully embracing LGTM ourselves but this is really interesting. Loki for us has been great, though, so what is better about CH other than maybe sql being more expressive than LogQL?

Re: Ten years of ClickHouse in open source

#84

Post by cynicalsecurity »

drchaim wrote: post #11

I discovered ClickHouse around 2017-18 and built a PoC to replace Elasticsearch: 5x better storage and qps, in a couple of weeks. Managers rejected it because it wasn't well known and was seen as "some database made by Russians." On a personal level, it's quite sad to have seen that train coming so early and not been able to get on board.

Supply chain risk.

Re: Ten years of ClickHouse in open source

#85

Post by nasretdinov »

adsharma wrote: post #52

It's interesting that the blog post places SQLite and Ladybird on the spectrum, but omits it's chief open source rival: DuckDB. Agree that Level 3 is what inspires confidence. But we need to invent new business models to sustain in the era of vibe-coded databases.

I think the main advantage of ClickHouse over DuckDB is *MergeTree family. It lets you sort data in the background, which allows for absurd levels of compression and performance when done right. ClickHouse can easily be 10x as performant as DuckDB querying Parquet when querying non-indexed columns, and obviously infinitely faster than DuckDB when you're touching primary key.

There are so many comparisons between the two, but realistically ClickHouse and DuckDB occupy completely separate niches, where DuckDB is just a really powerful analytics _engine_, and ClickHouse is a full database management system, with replication, MergeTree engine, etc.

Re: Ten years of ClickHouse in open source

#86

Post by nasretdinov »

ddorian43 wrote: post #9

Clickhouse is *really* gatekeeping the "zero copy replication" where you store data on object-storage and have high availability from the open source version.

That feature is central for ClickHouse Cloud offering which essentially is what lets others have (the rest of) ClickHouse for free. If you need that feature I think it's quite obvious you'll have to pay. I don't think it's an unreasonable stance at all.

P.S. I don't believe you are right to call it "zero copy" either. The object storage itself is replicated and definitely does copy and/or split data a lot under the hood. In some sense it's no different from ClickHouse's replication, apart from that ClickHouse can't use erasure codes to reduce the data footprint, whereas many advanced object stores can

Re: Ten years of ClickHouse in open source

#87

Post by ddorian43 »

nasretdinov wrote: post #86
ddorian43 wrote: post #9

Clickhouse is *really* gatekeeping the "zero copy replication" where you store data on object-storage and have high availability from the open source version.

That feature is central for ClickHouse Cloud offering which essentially is what lets others have (the rest of) ClickHouse for free. If you need that feature I think it's quite obvious you'll have to pay. I don't think it's an unreasonable stance at all. P.S. I don't believe you are right to call it "zero copy" either. The object storage itself is replicated and definitely does copy and/or split data a lot under the h…

It is called "zero copy" in clickhouse land.

That feature, is becoming central for an OLAP db, even a free/open-source one.

Re: Ten years of ClickHouse in open source

#88

Post by mial »

On top of being a really good OLAP database, what has been a game changer for me is the built-in connectors for bringing in data from remote sources. It can manage automatic recurring import of a s3 folder containing parquet/json files and it can also connect directly to Postgres. For our data warehouse at a medium sized newspaper, we switched from Druid+postgres+trino to just one big clickhouse node and I’ve never looked back. Much more performant, practical, and a lot less maintenance.

Re: Ten years of ClickHouse in open source

#89

Post by jaysh »

tills13 wrote: post #83
jaysh wrote: post #5

ClickHouse replacing Loki finally made our observability stack feel 'right'. It really is a powerhouse for logs and general analytical queries.

We are fully embracing LGTM ourselves but this is really interesting. Loki for us has been great, though, so what is better about CH other than maybe sql being more expressive than LogQL?

Off the top of my head:

- substantially better performance on the same hardware, even moreso for larger range queries (multiple days)

- no new query language to learn

- significantly more expressive as you said

- agents for scraping logs use way less CPU (I used to use grafana-agent which used about 80%, vector uses sub 5%)

- very intuitive to manage TTLs - I can keep some logs for 10 years, and some for 1 week based on the event in the JSON

- more compact storage, I didn't check scientifically but CH storage is better compressed at least 4-5x for us

- no running into maximum stream limits - struggled with these even on Grafana Cloud and didn't realise we silently lost a lot of logs

Honestly: why wouldn't you. Loki always felt like a mistake to me. A brand new query language, really counter-intuitive configuration, large ramp-up time for complex queries, lots of arguing about labels/cardinality etc. It all goes away when you drop it. I think logging should not be exotic or behave in unexpected ways.

Re: Ten years of ClickHouse in open source

#90

Post by byoj »

oulipo2 wrote: post #14
jaysh wrote: post #5

ClickHouse replacing Loki finally made our observability stack feel 'right'. It really is a powerhouse for logs and general analytical queries.

How do you use it for visualization? Do you use ClickStack? or something else?

We recently moved to openobserve for due to cost, but visualisations are good enough too.
Post reply on HN