Is ClickHouse Moving Away from Open Source? • Hacker News Board

Re: Is ClickHouse Moving Away from Open Source?

#31

Post by hodgesrm » Fri, Sep 22, 2023, 1:20 PM UTC

rjzzleep wrote: post #26

bitshiftin wrote: post #4

The blog author founded altinity. Altinity's main product offering is a hosted clickhouse service. The top 10 committers to clickhouse all seem to be clickhouse employees. Looking at altinity on github, they contribute much less open source. If clickhouse the company are spending 40%+ of their money to build the product, then others including altinity spend 5% dev and 80% marketing, they will get more customers. That…

Altinity maintains the ClickHouse operator which got a lot of people to use ClickHouse to begin with. ClickHouse has had a lot of corner cases that were reported from those kinds of people, myself included. If you look at some of the discussions, while a lot of the fixes come from the clickHouse team it would be unjust to say that the corner case discussions don't contribute to the fixes. I think part of the reason i…

To that point Altinity has contributed about 900 PRs to ClickHouse, and many more if you include ecosystem projects like the operator, clickhouse-backup (which we maintain), the community grafana plugin (over 11M downloads last I looked), ODBC driver, etc. All of this is open source.

We've also been very active on diagnosing problems, logging issues, and contributing ideas for solutions. Alexey Milovidov has logged the most issues of anyone (2376) but the next two people (1012, 810) are from Altinity. The #6 and #9 contributors of issues are also from Altinity.

Re: Is ClickHouse Moving Away from Open Source?

#32

Post by hodgesrm » Fri, Sep 22, 2023, 1:29 PM UTC

joshxyz wrote: post #28

hodgesrm wrote: post #24

Earlier quoted context omitted.

CEO of Altinity here, also editor of the blog article. It would be more interesting to address the points that it raises. If you are an open source user of ClickHouse, do you really want basic features like object storage for tables or ability to delete data efficiently withheld? This question is important regardless of who raises it. Projects like Kafka, Spark, PostgreSQL, and Kubernetes (among others) have solved i…

i appreciate your work here man, altinity even wrote a lot of useful blog posts about clickhouse back then, love those stuff. this whole thing reminds me of elastic and hashicorp, and it's hard to pick sides given that the core maintainers also worked their assess of building it in public, and the community contributors also put their effort into it. i think this common theme is unlocking a new era of software where…

Thanks. That's exactly why we wrote the article. Beyond any commercial considerations, we have worked with ClickHouse for many years and are personally invested in seeing it become the default analytic database worldwide. I believe the best path to that goal is robust support for open source development and distribution.

Re: Is ClickHouse Moving Away from Open Source?

#33

Post by snegussie » Fri, Sep 22, 2023, 1:36 PM UTC

Anyone familiar with Databend, Starrocks, or ByConity? They all focus on shared storage with separate compute. Currently checking out ByConity. Been using Clickhouse for quite a while and these were on my radar

Re: Is ClickHouse Moving Away from Open Source?

#34

Post by rjzzleep » Fri, Sep 22, 2023, 1:38 PM UTC

hodgesrm wrote: post #31

rjzzleep wrote: post #26

Earlier quoted context omitted.

Altinity maintains the ClickHouse operator which got a lot of people to use ClickHouse to begin with. ClickHouse has had a lot of corner cases that were reported from those kinds of people, myself included. If you look at some of the discussions, while a lot of the fixes come from the clickHouse team it would be unjust to say that the corner case discussions don't contribute to the fixes. I think part of the reason i…

To that point Altinity has contributed about 900 PRs to ClickHouse, and many more if you include ecosystem projects like the operator, clickhouse-backup (which we maintain), the community grafana plugin (over 11M downloads last I looked), ODBC driver, etc. All of this is open source. We've also been very active on diagnosing problems, logging issues, and contributing ideas for solutions. Alexey Milovidov has logged t…

I totally forgot about clickhouse-backup. It was frequently referenced in ClickHouse issues, and helped me and probably a lot of others understand how ClickHouse stores and transfers data.

Re: Is ClickHouse Moving Away from Open Source?

#35

Post by benjaminwootton » Fri, Sep 22, 2023, 2:06 PM UTC

lairv wrote: post #15

How do you chose between ClickHouse and DuckDB ? It feels like they solve the same problem

They solve the same problem in that they are OLAP data stores, but that's where the similarity ends. Clickhouse is a centralised OLAP store (like 10s of others) whilst DuckDB is an embedded database that is usually ran in process.

What is it about DuckDB and it's strange cult like following? It's nice that it's in process, but then it's an incremental improvement over Pandas. Nice tool and well implemented but I don't see what is transformative about it.

Re: Is ClickHouse Moving Away from Open Source?

#36

Post by jabart » Fri, Sep 22, 2023, 2:23 PM UTC

Clickhouse open source and cloud user. My understanding is that the cloud version uses S3. Which would mean that they have very specific tenant pattern and code to run that version. This may be why lightweight work a specific way in that environment, or they need a way to test it at scale that would be hard through a feature flag in the open source product. Lightweight deletes were released to both, and previous roadmaps listed updates as upcoming.

You should be using ReplacingMergeTree if you are doing updates at the current moment.

Re: Is ClickHouse Moving Away from Open Source?

#37

Post by unamedrus » Fri, Sep 22, 2023, 2:38 PM UTC

jabart wrote: post #36

Clickhouse open source and cloud user. My understanding is that the cloud version uses S3. Which would mean that they have very specific tenant pattern and code to run that version. This may be why lightweight work a specific way in that environment, or they need a way to test it at scale that would be hard through a feature flag in the open source product. Lightweight deletes were released to both, and previous road…

> Lightweight deletes were released to both

Because they were contributed by community member, not ClickHouse Inc core team.

https://github.com/ClickHouse/ClickHouse/pull/37893

Re: Is ClickHouse Moving Away from Open Source?

#38

Post by nnurmanov » Fri, Sep 22, 2023, 2:56 PM UTC

Here is a similar Thin-Crust Open Core model https://reactflow.dev/blog/asking-for-money-for-open-source/. But I hear CH and understand their move. I think OSS Sponsorship has failed; it does not generate enough money to pay a team of top engineers. The best move would have been to implement better paid support model and allow more users to pay for support. Currently CH charges immense amount for their support, so only large corporations could afford.

Re: Is ClickHouse Moving Away from Open Source?

#39

Post by hodgesrm » Fri, Sep 22, 2023, 2:58 PM UTC

jabart wrote: post #36

Clickhouse open source and cloud user. My understanding is that the cloud version uses S3. Which would mean that they have very specific tenant pattern and code to run that version. This may be why lightweight work a specific way in that environment, or they need a way to test it at scale that would be hard through a feature flag in the open source product. Lightweight deletes were released to both, and previous road…

> You should be using ReplacingMergeTree if you are doing updates at the current moment.

Indeed. Altinity and other community users like ContentSquare made numerous contributions to make it more usable. It's a promising approach to updates at scale and has improved markedly over the last few months.

That said you can't currently use RMT very efficiently in S3 because of overall limitations in MergeTree S3 table storage. We need to think about whether the improvements we're proposing will also enhance RMT. Thanks for bringing that up.

Re: Is ClickHouse Moving Away from Open Source?

#40

Post by jwatte » Fri, Sep 22, 2023, 3:19 PM UTC

nnurmanov wrote: post #38

Here is a similar Thin-Crust Open Core model https://reactflow.dev/blog/asking-for-money-for-open-source/ . But I hear CH and understand their move. I think OSS Sponsorship has failed; it does not generate enough money to pay a team of top engineers. The best move would have been to implement better paid support model and allow more users to pay for support. Currently CH charges immense amount for their support, so o…

But if they had a cheaper support model, the large corporations would also pay less, and they'd make even less money overall. Not to mention: support engineers are expensive, and researching tickets take a lot of time.