Earlier quoted context omitted.
You get high enough (hitting a metric with >100k unique labels), queries become unmanageable and incredibly slow when backed by the stock datastore (tsdb). However there are backing datastores (TimescaleDB, InfluxDB, VictoriaMetrics, etc) that ingest Prometheus metrics and enable higher cardinality.
This is where I fall off. Is Prometheus a DB that (can) forward data to another DB?
Systems Monitoring with Prometheus and Grafana
101–107 of 107 posts
Re: Systems Monitoring with Prometheus and Grafana
#102What's a good alternative to Prometheus when pulling stats is impractical? Say I want to monitor a personal laptop like I would a server. It will change networks and IP addresses, so pulling would be impractical to configure, whereas the laptop could easily(?) push its stats to a remote server.
If you use wireguard you can tell Prometheus to scrape your ip on the wireguard network
Re: Systems Monitoring with Prometheus and Grafana
#103We've got a somewhat similar landscape, on a pretty sizeable network - big investment in Zabbix and looking to move, perhaps slowly and perhaps only in part, towards Prometheus. Coming from a monitoring system that supports push and pull with elegant auto-discovery, we're struggling to work out a sane architecture around (effectively pull-only) Prometheus.
Take a look at VictoriaMetrics. It supports both pull and push models. It is inspired by Prometheus and it supports PromQL-inspired query language - MetricsQL [0]. [0] https://victoriametrics.github.io/MetricsQL.html
Yup - it's on our radar for evaluation.
Re: Systems Monitoring with Prometheus and Grafana
#104Earlier quoted context omitted.
I haven't spent much time on this but most of the docs were for setting it up on each hosts. Is there a proper tutorial for clusters? Also I wanted to keep the monitoring unaffected for other servers if one of them go down. If I setup a central server for monitoring then that becomes a single point of failure.
Grafana is meant to run as a single instance. For monitoring multiple servers, you need to get the metrics into one data store, from which Grafana will read. That's Prometheus' job. These pieces should not be on the same servers that run your product. For HA, you can run two or more Prometheuses as duplicates, so you can switch to another one if the main one is down.
Would the one datasource have a single database with several tables, one for each server? Lets say I am monitoring mysql. Currently I have a `mysql` table in a databse named `telegraf` on each host. Can I combine multiple influx datasources into a single dashboard beacause that would be easiler right now for my current setup?
Re: Systems Monitoring with Prometheus and Grafana
#105Earlier quoted context omitted.
My Prometheus system is a $10/mo Linode. It collects from 27 other hosts, and at least 100 services distributed across those hosts - doesn't even break a sweat. All the exporters run through a wireguard VPN. Prometheus is great for a small/medium SaaS type environment.
What do you use as a frontend? As far as I could tell grafana free tier doesn’t allow monitoring cluster of servers.
I use Grafana and some custom ones, I have only one Prometheus box so clustering is not a problem I'm having (and likely won't, I can vertical scale a long way for my smallish operation)
Re: Systems Monitoring with Prometheus and Grafana
#106Earlier quoted context omitted.
Grafana is meant to run as a single instance. For monitoring multiple servers, you need to get the metrics into one data store, from which Grafana will read. That's Prometheus' job. These pieces should not be on the same servers that run your product. For HA, you can run two or more Prometheuses as duplicates, so you can switch to another one if the main one is down.
Would the one datasource have a single database with several tables, one for each server? Lets say I am monitoring mysql. Currently I have a `mysql` table in a databse named `telegraf` on each host. Can I combine multiple influx datasources into a single dashboard beacause that would be easiler right now for my current setup?
You handle multiple servers by tagging your metrics with the server id or name. No need to create a table per server.
Each panel in a dashboard points to a specific data source, so you can have multiple data sources in one dashboard.
Re: Systems Monitoring with Prometheus and Grafana
#107Earlier quoted context omitted.
This is where I fall off. Is Prometheus a DB that (can) forward data to another DB?
Prometheus is a data format but it's also a "suite" of tools on top of that data format. Usually what happens is your app, db, whatever will expose metrics (http request status, average response time, etc) in the Prometheus format which is then scraped by the Prometheus ingestor. The ingestor stores those metrics in a (short-term) datastore called TSDB. Prometheus also ships with a little web UI as well that can quer…
Thanks for the explanation