Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grafana Mimir – Horizontally scalable long-term storage for Prometheus

grafana.com

Post reply on HN
Print view View original

51–60 of 121 posts

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#51

Post by pracucci »

sriv1211 wrote: post #48

What's the latency between sending a metric and being able to query it when using object storage (s3) instead of block storage? How do the transfer/retrieval (GET/PUT) costs factor in as well?

Good question! Grafana Mimir guarantees read-after-write. If a write request succeed, the metric samples you've written are guaranteed to be queried by any subsequent query.

Mimir employes write deamplification: it doesn't write immediately to the object storage but keeps most recently written data in-memory and/or local disk.

Mimir also employes several shared caches (supports Memcached) to reduce object storage (S3) access as much as possible.

You can learn more here in the Mimir architecture documentation: https://grafana.com/docs/mimir/latest/operators-guide/archit...

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#52

Post by AndyNemmity »

netingle wrote: post #21
halfmatthalfcat wrote: post #5

How does this stack up with https://github.com/thanos-io/thanos , which I've used to pretty good success. The only criticism I have of Thanos though was the amount of moving pieces to maintain.

(Tom here; I started the Cortex project on which Mimir is based and lead the team behind Mimir) Thanos is an awesome piece of software, and the Thanos team have done a great job building an vibrant community. I'm a big fan - so much so we used Thanos' storage in Cortex. Mimir builds on this and makes it even more scalable and performance (with a sharded compactor and query engine). Mimir is multitenant from day 1, wh…

Okay, but why? I am using Thanos today. It works, it's complex, when it breaks, it's a bit of a challenge to fix, but it happens. It doesn't break often.

It does the job. Mimir, which is based on Cortex, using either Mimir, or Cortex, what benefit am I getting?

I get asked every few months about moving off of Thanos to Cortex, and today now Mimir, and I don't have any substantial reason to do so. It feels like moving for the sake of moving.

I need to see some real reasoning as to why I am going to add value to move everything to Mimir.

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#53

Post by pracucci »

sciurus wrote: post #43
netingle wrote: post #25

Earlier quoted context omitted.

We tried to address this question on the Q&A blog post: https://grafana.com/blog/2022/03/30/qa-with-our-ceo-about-gr... It doesn't have to mean the end for Cortex, but others will have to step up to lead the project. We've tried to put other maintainers in place to kick start this.

I was going to ask what the migration path was from Cortex to Mimir, but I see you've documented that at https://grafana.com/docs/mimir/latest/migration-guide/migrat... . Thanks for the work you've done to make this easy.

This video also shows a live migration from Cortex to Mimir (running in Kubernetes): https://www.youtube.com/watch?v=aaGxTcJmzBw&ab_channel=Grafa...

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#55

Post by m1keil »

maccard wrote: post #45
m1keil wrote: post #42

Earlier quoted context omitted.

We are running Grafana and Prometheus on a single t3.xlarge instance with 150GB gp3 EBS. Excluding traffic, it costs ~ $100 USD per month. We are doing 10 second scrapes and currently have roughly 141k active time series. In Grafana Cloud it would cost... 15000 metrics for free. 126000/1000 * $8 = $882 Now here's the real kicker.. the pricing Grafana puts on their website are assuming 60 second scrape interval (1 dat…

> Excluding traffic, it costs ~ $100 USD per month I don't doubt that that's affordable, or cost competitive to AWS, but thats' about as cheap as you can do it, _and_ that's not including traffic. It's pretty much impossible to half that bill.

I excluded the traffic because the price is basically 0. This is internal traffic and a bunch of HTTP requests. It doesn't cost us $3000 a month.

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#57

Post by netingle »

AndyNemmity wrote: post #52
netingle wrote: post #21

Earlier quoted context omitted.

(Tom here; I started the Cortex project on which Mimir is based and lead the team behind Mimir) Thanos is an awesome piece of software, and the Thanos team have done a great job building an vibrant community. I'm a big fan - so much so we used Thanos' storage in Cortex. Mimir builds on this and makes it even more scalable and performance (with a sharded compactor and query engine). Mimir is multitenant from day 1, wh…

Okay, but why? I am using Thanos today. It works, it's complex, when it breaks, it's a bit of a challenge to fix, but it happens. It doesn't break often. It does the job. Mimir, which is based on Cortex, using either Mimir, or Cortex, what benefit am I getting? I get asked every few months about moving off of Thanos to Cortex, and today now Mimir, and I don't have any substantial reason to do so. It feels like moving…

Sounds like Thanos is working well for you, so in your position I wouldn't change anything.

There are a bunch of other reasons why people might choose Mimir; perhaps they have out grown some of the scalability limits, or perhaps they want faster high cardinality queries, or a different take on multi-tenancy.

Do remember Cortex (on which Mimir is based) predates Thanos as a project; Thanos was started to pursue a different architecture and storage concept. Thanos storage was clearly the way forward, so we adopted it. The architectures are still different: Thanos is "edge"-style IMO, Mimir is more centralised. Some people have a preference for one over the other.

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#58

Post by witcher »

halfmatthalfcat wrote: post #5

How does this stack up with https://github.com/thanos-io/thanos , which I've used to pretty good success. The only criticism I have of Thanos though was the amount of moving pieces to maintain.

(Bartek here: I co-started Thanos and maintain it with other companies)

Thanks for this - it's a good feedback. It's funny you mentioned that, because we actively try to reduce the number of running pieces e.g while we design our query sharding (parallelization) and pushdown features.

As Cortex/Mimir shows it's hard - if you want to scale out every tiny functionality of your system you end up with twenty different microservices. But it's an interesting challenge to have - eventually it comes to trade-offs we try to make in Thanos between simplicity, reliability and cost vs ultra max performance (Mimir/Cortex).

Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus

#59

Post by vineeth0297 »

MindTooth wrote: post #26

How does this compare to https://www.timescale.com/promscale I’m looking into choosing a backend for my metrics and always open for suggestions.

Hey!

Promscale PM here :)

Promscale is the open source observability backend for metrics and traces powered by SQL.

Whereas Mimir/Cortex is designed only for metrics.

Key differences:

1. Promscale is light in architecture as all you need is Promscale connector + TimescaleDB to store and analyse metrics, traces where as Cortex comes with highly scalable micro-services architecture this requires deploying 10's of services like ingestor, distributor, querier, etc.

2. Promscale offers storage for metrics, traces and logs (in future). One system for all observability data. whereas the Mimir/Cortex is purpose built for metrics.

3. Promscale supports querying the metrics using PromQL, SQL and traces using Jaeger query and SQL. whereas in Cortex/Mimir all you can use is PromQL for metrics querying.

4. The Observability data in Cortex/Mimir is stored in object store like S3, GCS whereas in Promscale the data is stored in relational database i.e. TimescaleDB. This means that Promscale can support more complex analytics via SQL but Cortex is better for horizontal scalability at really large scales.

5. Promscale offers per metric retention, whereas Cortex/Mimir offers a global retention policy across the metrics.

I hope this answers your question!

Post reply on HN