Live data from Hacker News

Building Observability with ClickHouse

cmtops.dev

21–26 of 26 posts

Re: Building Observability with ClickHouse

#21
post #14
post #7

Earlier quoted context omitted.

That's not truth. There are solutions for logging, which are very easy to setup and operate. For example, VictoriaLogs [1] (I'm its' author). It is designed from the grounds up to be easy to configure and use. It contains a single self-contained executable without external dependencies, which runs optimally on any hardware starting from Raspberry Pi and ending with a monster machine containing hundreds of CPU cores a…

Interesting, although the doc is not really user-friendly and doesn't show a lot of screenshots from the UI to get a sense of what the product can do

Yup, there is a big room for improvements there! BTW, VictoriaLogs docs are open source too [1], so we'll be more than happy receiving pull requests with improvements for the docs!

[1] https://github.com/VictoriaMetrics/VictoriaMetrics/tree/mast...

Re: Building Observability with ClickHouse

#22
post #5
post #2

I see lot of hype around ClickHouse these days. Few years ago I remember TimescaleDB making the rounds, arguably being predecessor for this sort of "observability on SQL" thinking. The article has short paragraph mentioning Timescale, but unfortunately it doesn't really go into comparing it to ClickHouse. How does HN see the situation these days, is ClickHouse simply overtaking Timescale on all axis? That sounds bit…

ClockHouse outperforms TimescaleDB in every aspect on large volumes of data. https://benchmark.clickhouse.com/ If you have small volumes of data (let's say less than a terabyte of data), then TimescaleDB is OK to use if you are OK with not so fast query performance.

> ClockHouse

That typo is a great mashup of TimescaleDB and ClickHouse.

Re: Building Observability with ClickHouse

#24
post #15

There is at least one basic factual error in this blog post, which makes me discount the whole thing. "But if you will use it, keep in mind that [InfluxDB] uses Bolt as its data backend." Simply not true. The author seems to have confused the storage that Raft consensus uses for metadata with that used for the time series data. InfluxDB has its own custom data storage layer for time series data, and has had so for ma…

Hey! The author of the article here.

Thanks for pointing out at this part of the text.

Please note that English is clearly not my native language. Sometimes, I may structure sentences in ambiguous ways due to grammatical or other errors in my writing. I'm not sure if this is the case.

To avoid further misinformation, let me elaborate what I meant to say in the paragraph you have mentioned. It's quoted below for other readers' comfort.

> But if you will use it, keep in mind that it uses Bolt as its data backend. The same Bolt used in the HashiCorp Vault integrated Raft storage and in etcd. Hence, it may need some maintenance too. More specifically, database compaction.

As per InfluxDB documentation, "InfluxDB uses BoltDB to store data including organization and user information, UI data, REST resources, and other key value data." (see here: https://docs.influxdata.com/influxdb/v2/reference/config-opt...). To me that's exactly what a data backend means. This piece of software uses different backends for different types of data if I got it right.

Since I haven't had any experience with InfluxDB itself, it's not clear to me whether its BoltDB storage may (or may not) need maintenance. In my opinion it's just a little detail that may be helpful for some people, so I mentioned it.

It's possible for me to redact this part of the text for more clarification, but that information by itself isn't very important to make me do that.

> There is at least one basic factual error in this blog post, which makes me discount the whole thing.

Everyone makes mistakes, that's why all of us filter and interpret information based on our experience accumulated through years of living on this planet. Such a radical change of perception to a large piece of information caused by a small mistake is not always necessary.

Again, thanks for being interested in making corrections! Have a great rest of your day!

Re: Building Observability with ClickHouse

#26

Earlier quoted context omitted.

What kinds of SQL queries could ClickHouse not handle? Were the limitations about expressivity of queries, performance, or something else? I'm considering using CH for storing observability (particularly tracing) data, so I'm curious about any footguns or other reasons it wouldn't be a good fit.

I'm editing the transcript right now, and he says it's more about exposing a nice API to the user. E.G: Clickhouse interval support, which is an important type for observability, was lacking. You couldn't subtract datetimes to get an interval. If you'd compared 2 milliseconds intervals to one second ones, it wouldn't look at the unit and would say 2 ms is bigger, etc. So he had to go to the dev team, and after enough…

One of the devs working on Logfire here. Part of it was the level of support. Like Samuel said the ClickHouse folks were not receptive to bug reports. The Timescale team is leagues ahead in that sense, they’re super responsive and helpful. Ultimately one of the reasons for choosing DataFusion was that it’s much more approachable of a project and indeed we’ve already gotten tremendous bidirectional benefit: the DataFusion team has helped us figure out some complex bits and we’ve done significant upstream contributions. By the way, DataFusion is now the fastest single node query engine on ClickBench: https://datafusion.apache.org/blog/2024/11/18/datafusion-fas...

Another reason we use DataFusion is multi-tenancy: we found it was hard to use RLS and such to implement multi-tenancy. We’ve had much better luck with the extensibility of DataFusion.

Post reply on HN