Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grafana Mimir – Horizontally scalable long-term storage for Prometheus

grafana.com

Post reply on HN
Print view View original

111–120 of 121 posts

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#111

Post by _msw_ »

sciurus wrote: post #49
nosequel wrote: post #8

Grafana Labs needs to make a convincing comparison chart of some kind between Mimir, Thanos, and Cortex. Thanos and Cortex are both mature projects and are both CNCF Incubating projects. Why would anyone switch to a new prometheus long-term storage solution from those? *EDIT* : I see from another reply there is a basic comparison to Cortex here: https://grafana.com/blog/2022/03/30/announcing-grafana-mimir... To the M…

It looks like this is a fork of Cortex driven by the maintainers employed by Grafana Labs, done so they can change the license to one that will prevent cloud providers like Amazon from offering it without contributing changes back. This is interesting, since Amazon offers both hosted Grafana and Cortex today. I was under the impression Amazon and Grafana Labs were successfully collaborating (unlike e.g. AWS and Elast…

Disclosure: I work for AWS, but I don't work on the Amazon Managed Service for Prometheus. I have my own very long held opinions about Free and Open Source software, and I am only speaking for myself.

To me, the AGPLv3 license isn't about forcing software users to "give changes back" to a project. It is about giving the permissions to users of software that are necessary for Software Freedom [1] when they access a program over a network. In practice, that means that changes often flow "upstream" to copyleft licensed programs one way or another. But it was never about obligating changes to be "given back" to upstream. In my personal opinion, you should be "free to fork" Free and Open Source Software (FOSS). Indeed, the Grafana folks seem to have decided to do that with Grafana Mimir.

Personally, I hope that they accept contributions under the AGPLv3 license, and hold themselves to the same obligations that others are held to with regard to providing corresponding source code of derivative works when it is made available to users over a network. In my personal opinion, too often companies use a contributor agreement that excuses them from those obligations, and also allows them to sell the software to others under licenses that do not carry copyleft obligations. See [2] for a blog post that goes into some detail about this.

If you look at the Coretex project MAINTAINERS file [3], you will see that there are two folks listed that currently work at AWS, but no other company other than Grafana Labs today. I would love to see more diversity in maintainers for a project like this, as I think too many maintainers from any one company isn't the best for long term project sustainability.

I think if you look at the Cortex Community Meeting minutes [4], you can see that AWS folks are regularly "showing up" in healthy numbers, and working collaboratively with anyone who accepts the open invitation to participate. There have been some pretty big improvements to Coretex that have merged lately, like some of the work on parallel compaction [5, 6].

TL;DR, I think it is easy to jump to some conclusions about how things are going in a FOSS project that don't hold water if you do some cursory exploration. I think best way to know what's going on in a project is to get involved!

--

[1] the rights needed to: run the program for any purpose; to study how the program works, and modify it; to redistribute copies; to distribute copies of modified versions to others

[2] https://meshedinsights.com/2021/06/14/legally-ignoring-the-l...

[3] https://github.com/cortexproject/cortex/blob/master/MAINTAIN...

[4] https://docs.google.com/document/d/1shtXSAqp3t7fiC-9uZcKkq3m...

[5] https://aws.amazon.com/blogs/opensource/scaling-cortex-with-...

[6] https://github.com/cortexproject/cortex/pull/4624

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#112

Post by notacoward »

vladvasiliu wrote: post #75
notacoward wrote: post #37

Earlier quoted context omitted.

Multi-tenancy is something that shouldn't be underestimated. A lot of people think it's just a checklist item until (a) they need it or (b) they try to implement it in an existing system. Kudos for making it a day-one feature.

While I agree with your point in the general case, would you mind elaborating on the specific case of Prometheus? My understanding is that the recommended best-practice for Prometheus is to deploy as many of them as necessary, as close to the monitored infrastructure as possible. What use case would require deploying a single Mimir, so supposedly Prometheus (cluster) in the case of serving multiple tenants? Why not j…

I don't know Prometheus, but I would imagine the answer depends on just how many clients you have. Probably doesn't matter if you're talking just a few. If it's a lot, then separate instances can be very expensive in terms of operational complexity and waste due to resource fragmentation. Multi-tenancy is good for bringing both of those back under control. Is there something about Prometheus that would negate that?

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#113

Post by vladvasiliu »

notacoward wrote: post #112
vladvasiliu wrote: post #75

Earlier quoted context omitted.

While I agree with your point in the general case, would you mind elaborating on the specific case of Prometheus? My understanding is that the recommended best-practice for Prometheus is to deploy as many of them as necessary, as close to the monitored infrastructure as possible. What use case would require deploying a single Mimir, so supposedly Prometheus (cluster) in the case of serving multiple tenants? Why not j…

I don't know Prometheus, but I would imagine the answer depends on just how many clients you have. Probably doesn't matter if you're talking just a few. If it's a lot , then separate instances can be very expensive in terms of operational complexity and waste due to resource fragmentation. Multi-tenancy is good for bringing both of those back under control. Is there something about Prometheus that would negate that?

For one, it doesn't really support authentication (although it's on the roadmap).

I'm no Prometheus expert, but since you're pretty much expected to be running a bunch of servers anyway, the operational complexity has to be handled even for just one client.

You do have a point on resource fragmentation, but IME Prometheus' resource usage is fairly predictable, so you could probably mitigate that to a point.

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#114

Post by deepsun »

smw wrote: post #109
nosequel wrote: post #8

Grafana Labs needs to make a convincing comparison chart of some kind between Mimir, Thanos, and Cortex. Thanos and Cortex are both mature projects and are both CNCF Incubating projects. Why would anyone switch to a new prometheus long-term storage solution from those? *EDIT* : I see from another reply there is a basic comparison to Cortex here: https://grafana.com/blog/2022/03/30/announcing-grafana-mimir... To the M…

Seems like people should throw VictoriaMetrics into comparisons like this, as well?

Yea, although making benchmarks properly is no easy task, and can be pretty time consuming, especially if you involve all the contestants for fairness. They are not interested in releasing a benchmark if they don't look good in it.

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#115

Post by valyala »

jaigupta wrote: post #108

This is about Prometheus but Mimir makes it interesting. I can't find any other open source time series database except Mimir/Cortex which allows this much scale (clustering options in their open source version). Our use case will have high cardinality and Mimir seems to fit very well. Can we use Prometheus/Mimir as general purpose time series database? Prometheus is built for monitoring purposes and may not be for g…

> I can't find any other open source time series database except Mimir/Cortex which allows this much scale (clustering options in their open source version)

The following open source time series databases also can scale horizontally to many nodes:

- Thanos - https://github.com/thanos-io/thanos/

- M3 - https://github.com/m3db/m3

- Cluster version of VictoriaMetrics - https://docs.victoriametrics.com/Cluster-VictoriaMetrics.htm... (I'm CTO at VictoriaMetrics)

> Can we use Prometheus/Mimir as general purpose time series database?

This depends on what do you mean under "general purpose time series database". Prometheus/Mimir are optimized for storing (timestamp, value) series where timestamp is a unix timestamp in milliseconds and value is a floating-point number. Each series has a name and can have arbitrary set of additional (label=value) labels. Prometheus/Mimir aren't optimized for storing and processing series of other value types such as strings (aka logs) and complex datastructures (aka events and traces).

So, if you need storing time series with floating-point values, then Prometheus/Mimir may be a good fit. Otherwise take a look at ClickHouse [1] - it can efficiently store and process time series with values of arbitrary types.

[1] https://clickhouse.com/

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#116

Post by valyala »

cip01 wrote: post #94
valyala wrote: post #84

Earlier quoted context omitted.

It would be great if you could provide a few practical examples for "Prometheus remote-read" protocol given its' restrictions [1]. [1] https://github.com/prometheus/prometheus/issues/4456

Which restrictions do you have in mind? Quick look at the issue looks like it wanted to avoid using local storage by Prometheus, but that’s Prometheus specific problem, not remote-read problem. Remote-read is a generic protocol ( https://github.com/prometheus/prometheus/blob/a1121efc18ba15... ), you pass query (start/end time and matchers), and get back data.

> Which restrictions do you have in mind?

You wrote in the previous comment:

> ... so external systems (eg Prometheus) can read data from them easily.

I pointed to an issue, which prevents from practical usage of remote read protocol from Prometheus itself.

As for the interoperability with external systems, Prometheus querying API [1] is better suited for this task than Prometheus remote read protocol because of the following reasons:

- Prometheus querying API is easy to use, since it is just JSON over HTTP (unlike compressed protobuf used for Prometheus remote read). E.g. humans can test and debug it either directly in web-browser or in a command-line shell with curl.

- Prometheus querying API is already supported by popular external systems such as Grafana.

- Many Prometheus-compatible systems such as Thanos, Cortex, M3, VictoriaMetrics, etc. support Prometheus querying API out of the box.

[1] https://prometheus.io/docs/prometheus/latest/querying/api/

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#117

Post by jaigupta »

valyala wrote: post #115
jaigupta wrote: post #108

This is about Prometheus but Mimir makes it interesting. I can't find any other open source time series database except Mimir/Cortex which allows this much scale (clustering options in their open source version). Our use case will have high cardinality and Mimir seems to fit very well. Can we use Prometheus/Mimir as general purpose time series database? Prometheus is built for monitoring purposes and may not be for g…

> I can't find any other open source time series database except Mimir/Cortex which allows this much scale (clustering options in their open source version) The following open source time series databases also can scale horizontally to many nodes: - Thanos - https://github.com/thanos-io/thanos/ - M3 - https://github.com/m3db/m3 - Cluster version of VictoriaMetrics - https://docs.victoriametrics.com/Cluster-VictoriaMe…

I meant all Prometheus based solutions, includes Thanos, M3, VictoriaMetrics. Thank you for your answer.

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#118

Post by tarun_anand »

vineeth0297 wrote: post #86
tarun_anand wrote: post #65

Earlier quoted context omitted.

Thanks... how do we do reporting/dashboards/alerts with Promscale? Also, any performance benchmarks?

Promscale supports reporting/ingestion of data using Prometheus remote-write for metrics, OTLP (OpenTelemetry Line Protocol) for traces. Dashboards you can use Promscale as Prometheus datasource for PromQL based querying, visualising, as Jaeger datasource for querying, visualising traces and as PostgreSQL datasource to query both metrics and traces using SQL. If you are interested in visualising data using SQL, we re…

Thanks... I will try it out. What we really need is SQL based/OTel based systems. It makes life so much easier.

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#119

Post by vineeth0297 »

tarun_anand wrote: post #118
vineeth0297 wrote: post #86

Earlier quoted context omitted.

Promscale supports reporting/ingestion of data using Prometheus remote-write for metrics, OTLP (OpenTelemetry Line Protocol) for traces. Dashboards you can use Promscale as Prometheus datasource for PromQL based querying, visualising, as Jaeger datasource for querying, visualising traces and as PostgreSQL datasource to query both metrics and traces using SQL. If you are interested in visualising data using SQL, we re…

Thanks... I will try it out. What we really need is SQL based/OTel based systems. It makes life so much easier.

Feel free to reach out to us in Timescale community slack :), Would love to help you in getting started with Promscale!

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#120

Post by justincormack »

_msw_ wrote: post #111
sciurus wrote: post #49

Earlier quoted context omitted.

It looks like this is a fork of Cortex driven by the maintainers employed by Grafana Labs, done so they can change the license to one that will prevent cloud providers like Amazon from offering it without contributing changes back. This is interesting, since Amazon offers both hosted Grafana and Cortex today. I was under the impression Amazon and Grafana Labs were successfully collaborating (unlike e.g. AWS and Elast…

Disclosure: I work for AWS, but I don't work on the Amazon Managed Service for Prometheus. I have my own very long held opinions about Free and Open Source software, and I am only speaking for myself. To me, the AGPLv3 license isn't about forcing software users to "give changes back" to a project. It is about giving the permissions to users of software that are necessary for Software Freedom [1] when they access a pr…

Their other AGPL projects all have a CLA and they state you can buy them as part of Grafana Enterprise without the AGPL license https://grafana.com/blog/2022/03/30/qa-with-our-ceo-about-gr... so they are not offering symmetric terms to themselves.
Post reply on HN