[deleted]
Grafana Mimir – Horizontally scalable long-term storage for Prometheus
41–50 of 121 posts
Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus
#42Earlier quoted context omitted.
Recently switched from their cloud service back to on-premise. The cloud version wasn't being updated and the entire setup experience left a lot to be desired with how you connect their on-premise grafana agent, especially if you aren't using their easy button deployment stuff. Also, billing for metrics is insane, as on any given day my metric load may vary between 5-7k or more. This caused some operational overhead…
> For $50/mo, you can self host everything easier, cheaper and with more control IMO. Can you give an example as to how you could self host a grafana stack for $50/month? On AWS that buys you 4 cores, 8GB memory and 0 storage, and it's certainly not easier than clicking one button on the grafana website.
We are running Grafana and Prometheus on a single t3.xlarge instance with 150GB gp3 EBS.
Excluding traffic, it costs ~ $100 USD per month.
We are doing 10 second scrapes and currently have roughly 141k active time series. In Grafana Cloud it would cost...
15000 metrics for free. 126000/1000 * $8 = $882
Now here's the real kicker.. the pricing Grafana puts on their website are assuming 60 second scrape interval (1 data point/minute or DPM). If you are doing 6 DPM, that's $8 * 6 per 1000 time series!
So final bill.. drum rolls
126000/1000 * $8 * 6 = $6048
Yes. That's a 60x.
Now, sure, we don't get the scale, the backups, the SLA.. but we can live without it. And when Prometheus will start acting slowly, we will just bump it to t3.2xl, or spend some time and filter out some of the noisy metrics we might have around.
Btw, if you try to find any information about what is a "time series" or a "metric" on the Grafana's pricing page, good luck.
https://grafana.com/docs/grafana-cloud/metrics-control-usage...
Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus
#43Sad news for Cortex, with most of the maintainer moving on to Mimir, I fear it's pretty much dead in the water.
We tried to address this question on the Q&A blog post: https://grafana.com/blog/2022/03/30/qa-with-our-ceo-about-gr... It doesn't have to mean the end for Cortex, but others will have to step up to lead the project. We've tried to put other maintainers in place to kick start this.
I was going to ask what the migration path was from Cortex to Mimir, but I see you've documented that at https://grafana.com/docs/mimir/latest/migration-guide/migrat... . Thanks for the work you've done to make this easy.
Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus
#44Grafana Labs needs to make a convincing comparison chart of some kind between Mimir, Thanos, and Cortex. Thanos and Cortex are both mature projects and are both CNCF Incubating projects. Why would anyone switch to a new prometheus long-term storage solution from those? *EDIT* : I see from another reply there is a basic comparison to Cortex here: https://grafana.com/blog/2022/03/30/announcing-grafana-mimir... To the M…
I agree! Which is why I put one in the blog post ;-) https://grafana.com/blog/2022/03/30/announcing-grafana-mimir...
I'm not seeing a comparison to Thanos
Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus
#45Earlier quoted context omitted.
> For $50/mo, you can self host everything easier, cheaper and with more control IMO. Can you give an example as to how you could self host a grafana stack for $50/month? On AWS that buys you 4 cores, 8GB memory and 0 storage, and it's certainly not easier than clicking one button on the grafana website.
We are running Grafana and Prometheus on a single t3.xlarge instance with 150GB gp3 EBS. Excluding traffic, it costs ~ $100 USD per month. We are doing 10 second scrapes and currently have roughly 141k active time series. In Grafana Cloud it would cost... 15000 metrics for free. 126000/1000 * $8 = $882 Now here's the real kicker.. the pricing Grafana puts on their website are assuming 60 second scrape interval (1 dat…
> Excluding traffic, it costs ~ $100 USD per month
I don't doubt that that's affordable, or cost competitive to AWS, but thats' about as cheap as you can do it, _and_ that's not including traffic. It's pretty much impossible to half that bill.
Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus
#46Earlier quoted context omitted.
It’s a centralised multi-tenant store, supporting the Prometheus query API. So you can point clients directly at Mimir, they send in PromQL and they get data back in Json. (Note I work on Mimir)
But who does the scraping of the prometheus agents? Mimir or still prometheus server?
Last year I wrote a blog post about this exact question: Who watches the watchers?
The general takeaway is that you run a minimal prometheus/alertmanager setup that only scrapes the agents, then use a dead man switch-like system to ensure this pipeline keeps working.
Link: https://grafana.com/blog/2021/04/08/how-we-use-metamonitorin...
Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus
#47Earlier quoted context omitted.
Recently switched from their cloud service back to on-premise. The cloud version wasn't being updated and the entire setup experience left a lot to be desired with how you connect their on-premise grafana agent, especially if you aren't using their easy button deployment stuff. Also, billing for metrics is insane, as on any given day my metric load may vary between 5-7k or more. This caused some operational overhead…
> For $50/mo, you can self host everything easier, cheaper and with more control IMO. Can you give an example as to how you could self host a grafana stack for $50/month? On AWS that buys you 4 cores, 8GB memory and 0 storage, and it's certainly not easier than clicking one button on the grafana website.
> $50/month? On AWS that buys you 4 cores, 8GB memory and 0 storage
Self-hosting on AWS is kind of counterproductive. Look into "cloud" metal servers and the money will go much further.
Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus
#48What's the latency between sending a metric and being able to query it when using object storage (s3) instead of block storage?
How do the transfer/retrieval (GET/PUT) costs factor in as well?
Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus
#49Grafana Labs needs to make a convincing comparison chart of some kind between Mimir, Thanos, and Cortex. Thanos and Cortex are both mature projects and are both CNCF Incubating projects. Why would anyone switch to a new prometheus long-term storage solution from those? *EDIT* : I see from another reply there is a basic comparison to Cortex here: https://grafana.com/blog/2022/03/30/announcing-grafana-mimir... To the M…
It looks like this is a fork of Cortex driven by the maintainers employed by Grafana Labs, done so they can change the license to one that will prevent cloud providers like Amazon from offering it without contributing changes back.
This is interesting, since Amazon offers both hosted Grafana and Cortex today. I was under the impression Amazon and Grafana Labs were successfully collaborating (unlike e.g. AWS and Elastic), but seems like that's not the case.
Re: Grafana Mimir – Horizontally scalable long-term storage for Prometheus
#50More engineering effort going into reinventing things that already exist to upsell people on Grafana cloud. What about focusing on the core value that Grafana provides, dashboards? Grafana 8 alerting is still in my opinion at a beta level. Dashboards as code has made no meaningful progress outside of community attempts in the past 3 years. The documentation for Grafana 8 alerts is still subpar. All of these things as…
Understandable critique, but I absolutely love a lot of Grafana’s redundant offerings. For example, operationally speaking it is drastically simpler to set up a scalable Grafana Tempo instance than Jaeger, in my opinion. Grafana offering competent object storage backends for their software has made them dramatically easier to operate and maintain.
That’s also another thing: a decent amount of Grafana software (Mimir, Loki, Tempo…) are OSS, so while they definitely are using those softwares in their paid offering, they absolutely still benefit OSS users. I’m messing with Tempo for telemetry in my (admittedly embarrassingly weak) home lab endeavors and it’s pretty cool.