We Built a 19 PiB Logging Platform with ClickHouse and Saved Millions
1–10 of 13 posts
Re: We Built a 19 PiB Logging Platform with ClickHouse and Saved Millions
#2Hey! I'm the original author of this post. I'm so excited to share our journey with ClickHouse and the open source Observability world, I'll be happy to answer any questions you may have!
Re: We Built a 19 PiB Logging Platform with ClickHouse and Saved Millions
#3Great write up.
>The recent efforts to move the JSON type to production-ready status will be highly applicable to our logging use case. This feature is currently being rearchitected, with the development of the Variant type providing the foundation for a more robust implementation. When ready, we expect this to replace our map with more strongly typed (i.e. not uniformly typed) metadata structures that are also possibly hierarchical.
Very happy to see ClickHouse dogfooding itself for storing logs - hope this will help to hasten the work on improving the the json type more suitable to dynamic documents.
Re: We Built a 19 PiB Logging Platform with ClickHouse and Saved Millions
#4Great write up. >The recent efforts to move the JSON type to production-ready status will be highly applicable to our logging use case. This feature is currently being rearchitected, with the development of the Variant type providing the foundation for a more robust implementation. When ready, we expect this to replace our map with more strongly typed (i.e. not uniformly typed) metadata structures that are also possi…
Yes, we are working on it! :) Taking some of the learnings from current experimental JSON Object datatype, we are now working on what will become the production-ready implementation. Details here: https://github.com/ClickHouse/ClickHouse/issues/54864
Variant datatype is already available as experimental in 24.1, Dynamic datatype is WIP (PR almost ready), and JSON datatype is next up. Check out the latest comment on that issue with how the Dynamic datatype will work: https://github.com/ClickHouse/ClickHouse/issues/54864#issuec...
Re: We Built a 19 PiB Logging Platform with ClickHouse and Saved Millions
#5Hey! I'm the original author of this post. I'm so excited to share our journey with ClickHouse and the open source Observability world, I'll be happy to answer any questions you may have!
Very cool write up. I'm curious about any challenges you had using Grafana? Also, do you think this sort of system would work as an alternative to Splunk as well?
Re: We Built a 19 PiB Logging Platform with ClickHouse and Saved Millions
#6Its a log. When a log is that big, is it still use full?
Re: We Built a 19 PiB Logging Platform with ClickHouse and Saved Millions
#7The use-case of 19 PiB of logging data feels very constructed to me. I worked for smaller and bigger companies and never faced logging into the petabyte range. I'm not saying it's not a thing, FAANG level companies certainly have such needs, but they have their own large scale solutions already. The question remains, besides bragging, who is the average Joe with 19 PiB of logging data you might want to address as potential customer?
What would be useful from my perspective are benchmarks in the more common terabyte range. How much faster is it to query compared to existing cloud offering, what features does e.g. Datadog vs Clickhouse has to analyze the data? In the end the raw data is not much useful if you cannot easily find and extract meaningful data out of it.
Re: We Built a 19 PiB Logging Platform with ClickHouse and Saved Millions
#8Hey! I'm the original author of this post. I'm so excited to share our journey with ClickHouse and the open source Observability world, I'll be happy to answer any questions you may have!
Very cool write up. I'm curious about any challenges you had using Grafana? Also, do you think this sort of system would work as an alternative to Splunk as well?
Grafana works really well out of the box for most use-cases. Initially the LogHouse UI was built using the out-of-the-box Grafana tools (along with the ClickHouse data source plugin which we maintain)
You can get a really long way with the zero code dashboarding tools, especially with the latest plugin release (4.0) which comes with a completely rebuilt query builder which has an opinionated mode specifically for OpenTelemetry data. LogHouse exhausted the limits of the zero-code UI in a few places and it was necessary to evolve the UI from a dashboard into a Grafana Plugin. Doing so gives you much more control to build a full application using the Grafana primitives. I really like that as an SRE I can build a webapp without spending any time building UI components, instead just controlling the layout on page and declaring “this panel has the following `SELECT…` query” which is generated by some typescript function.
Examples of things which we can do on top of Scenes are:
- Pick a different schema based on the query parameters. For instance we have different schemas for different applications (Keeper/Server/Generic K8s app) and the app picks the necessary schema
- Always show the full generated SQL query on the page (We like to use Grafana UI to start off and then jump into fully manual SQL for deeper analysis)
- Take one filter value (for instance, k8s namespace) and look up all of the other filters required (pod names which were live during the time period, region, cell ect.)
- Some small gadgets like enabling users to import the time range from another application URL like DataDog. Oftentimes we start by looking at metrics in another source and then want to jump into the logs.
Re: We Built a 19 PiB Logging Platform with ClickHouse and Saved Millions
#9The use-case of 19 PiB of logging data feels very constructed to me. I worked for smaller and bigger companies and never faced logging into the petabyte range. I'm not saying it's not a thing, FAANG level companies certainly have such needs, but they have their own large scale solutions already. The question remains, besides bragging, who is the average Joe with 19 PiB of logging data you might want to address as pot…
We explored a scale similar to what you described in another blog:
https://clickhouse.com/blog/cost-predictable-logging-with-cl...
Feature differentiation is actually a pretty interesting topic for o11y. You can do many things with an OLAP store but you need to be aware of the differences with of the shelf solutions, I try to summarize it here: https://clickhouse.com/blog/the-state-of-sql-based-observabi...
I hope this helps! I'd love to hear your opinion about it
Re: We Built a 19 PiB Logging Platform with ClickHouse and Saved Millions
#10The use-case of 19 PiB of logging data feels very constructed to me. I worked for smaller and bigger companies and never faced logging into the petabyte range. I'm not saying it's not a thing, FAANG level companies certainly have such needs, but they have their own large scale solutions already. The question remains, besides bragging, who is the average Joe with 19 PiB of logging data you might want to address as pot…
We explored a scale similar to what you described in another blog: https://clickhouse.com/blog/cost-predictable-logging-with-cl... Feature differentiation is actually a pretty interesting topic for o11y. You can do many things with an OLAP store but you need to be aware of the differences with of the shelf solutions, I try to summarize it here: https://clickhouse.com/blog/the-state-of-sql-based-observabi... I hope th…
Thanks, this looks indeed much more interesting. I'm definitely looking into this.