Interesting post. However, I believe that most content, and especially broad technical one like this, absolutely needs a balanced amount of relevant visual elements (e.g., images, diagrams). If you want it to be readable, that is.
Systems Monitoring with Prometheus and Grafana
51–60 of 107 posts
Re: Systems Monitoring with Prometheus and Grafana
#52Interesting post. However, I believe that most content, and especially broad technical one like this, absolutely needs a balanced amount of relevant visual elements (e.g., images, diagrams). If you want it to be readable, that is.
Yeah, odd post that talks a lot about Grafana and visualizations then uses absolutely none.
Re: Systems Monitoring with Prometheus and Grafana
#53Interesting post. However, I believe that most content, and especially broad technical one like this, absolutely needs a balanced amount of relevant visual elements (e.g., images, diagrams). If you want it to be readable, that is.
Yeah, odd post that talks a lot about Grafana and visualizations then uses absolutely none.
Yes, it's quite ironic. Hopefully, the authors will eventually improve the post, because core content is valuable.
Re: Systems Monitoring with Prometheus and Grafana
#54Earlier quoted context omitted.
Doesn’t it also have cardinality issues?
You get high enough (hitting a metric with >100k unique labels), queries become unmanageable and incredibly slow when backed by the stock datastore (tsdb). However there are backing datastores (TimescaleDB, InfluxDB, VictoriaMetrics, etc) that ingest Prometheus metrics and enable higher cardinality.
And M3DB too if you want to cluster and scale out, vs cloud store
Re: Systems Monitoring with Prometheus and Grafana
#55Earlier quoted context omitted.
We've used Thanos to aggregate multiple Prometheus (Promethii?) across our clusters to enable us to scale, each Prometheus deals only with a subset of scrape targets. Biggest issue I've had was an app that was accidentally publishing several thousand metrics which caused the default scrape timeout of 15s to kick in. (It was publishing Kafka lag per consumer group per topic, which was fine and dandy, until someone rel…
>Promethii Prometheuses. ii is for latin words. Prometheus is/was Greek. I guess you could use Prometheoí but it would quickly derail any conversation. :)
There was a talk on Promcon 2016 about this subject [1]. The conclusion was: in English, indeed, Prometheuses. In Ancient Greek: Prometheis.
Re: Systems Monitoring with Prometheus and Grafana
#56I have a love/hate relationship with Prometheus. If I had no budget for metrics its likely the thing I would reach for, but I’m dying for someone to open source a ‘next level’ metrics system (something like Monarch or Circonus but free). But woe betide the team that has to run it as a service. Not that other metrics systems are better but Prometheus can be brutal in that space. As a ‘squad level’ tool its really good…
M3 is meant to be an open source, central, horizontally scalable metrics store - but your mileage may vary. Either way, check it out: https://m3db.io
Re: Systems Monitoring with Prometheus and Grafana
#57Grafana truly is best in class, but I have strong reservations about Prometheus. I really want to like it, it’s just so _easy_, publish a little webpage with your metrics and Prometheus takes care of the rest. Lovely. But I often find that the cardinality of the data is substantially lower than even the defaults of alternatives (influxdb has 1s and even Zabbix has 5s). Not to mention the lost writes (missing data poi…
My understanding is that Prometheus is designed for you to deploy multiple instances within your company, rather than deploying a limited number of instances for the company or division. So I would reasonably run a Prometheus instance by myself or with my neighboring teams rather than depending on a centralized instance run by $OPS.
Re: Systems Monitoring with Prometheus and Grafana
#58Earlier quoted context omitted.
Doesn’t it also have cardinality issues?
There's no real way to develop a metrics system without cardinality issues. Where you draw the line depends on the backing database but they're all fairly constrained.
Honestly, it’s incredibly impressive how far you can push them as is. We send a lot of data into these systems.
Re: Systems Monitoring with Prometheus and Grafana
#59Prometheus and Grafana are awesome, use them personally for all my monitoring. However I’m still trying to nail down my high cardinality/highly unique metrics-like data story. What are people using? I’ve heard a combination of Cassandra/BigTable and Spark as a potential solution?
I'd be curious to hear if anyone has done serious evaluation of high-cardinality use-cases of Victoriametrics.
I went from an Influx getting crushed to VM running in a container with 1/8th the resources and it works fine, 1.5m active cardinality. Could handle a lot more probably. Auto fill in Grafana breaks but oh well!
Re: Systems Monitoring with Prometheus and Grafana
#60Does anyone have anything good or bad to share about using Grafana as a front end for metrics logged in AWS cloudwatch? I know it has a plug in and I'm fed up with how bad the cloudwatch dashboards are so wondering if I should check it out.
Well, I’d say give it a try. We are using CloudWatch as a Grafana datasource, because this way you can concentrate more of your monitoring to one place, which is useful during troubleshooting. With Grafana 7.x, you can even check and correlate your CloudWatch logs inside Grafana, deeplinked to the AWS Console. After this major version, you can even wire Jaeger into Grafana, so you have a one stop solution for tracing as well (and logging, if you utilize Loki too).