Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Systems Monitoring with Prometheus and Grafana

flightaware.engineering

Post reply on HN
Print view View original

71–80 of 107 posts

Re: Systems Monitoring with Prometheus and Grafana

#71

Post by dewey »

Legogris wrote: post #31
dijit wrote: post #14

Grafana truly is best in class, but I have strong reservations about Prometheus. I really want to like it, it’s just so _easy_, publish a little webpage with your metrics and Prometheus takes care of the rest. Lovely. But I often find that the cardinality of the data is substantially lower than even the defaults of alternatives (influxdb has 1s and even Zabbix has 5s). Not to mention the lost writes (missing data poi…

Victoriametrics[0] is API-compatible with Prometheus but also a horizontally scalable, distributed and persisting timeseries database (cf influxdb). Together with vmagent it essentially becomes a HA drop-in replacement (almost) for Prometheus. [0]: https://github.com/VictoriaMetrics/VictoriaMetrics

These might be a good read if you are considering it:

- https://www.robustperception.io/evaluating-performance-and-c...

- https://medium.com/@valyala/evaluating-performance-and-corre...

Re: Systems Monitoring with Prometheus and Grafana

#72

Post by dewey »

abhishekjha wrote: post #47
edoceo wrote: post #5

Earlier quoted context omitted.

My Prometheus system is a $10/mo Linode. It collects from 27 other hosts, and at least 100 services distributed across those hosts - doesn't even break a sweat. All the exporters run through a wireguard VPN. Prometheus is great for a small/medium SaaS type environment.

What do you use as a frontend? As far as I could tell grafana free tier doesn’t allow monitoring cluster of servers.

You could self host it.

Re: Systems Monitoring with Prometheus and Grafana

#73

Post by michaelmcdonald »

abhishekjha wrote: post #46
molecule wrote: post #44

Earlier quoted context omitted.

Telegraf + InfluxDB?

This is my setup on all my raspberrypis. I have not be been able to figure out how to monitor a cluster though. I saw that grafana free tier doesn’t allow a cluster of servers getting monitored. I have telegraf + influxdb + grafana installed on all my servers.

Could you expand on what you mean by:

> grafana free tier doesn’t allow a cluster of servers getting monitored.

Is there a particular aspect of the cluster you're missing? Is it that you don't want individual server metrics?

Re: Systems Monitoring with Prometheus and Grafana

#74

Post by goranerik »

Jedd wrote: post #26

We've got a somewhat similar landscape, on a pretty sizeable network - big investment in Zabbix and looking to move, perhaps slowly and perhaps only in part, towards Prometheus. Coming from a monitoring system that supports push and pull with elegant auto-discovery, we're struggling to work out a sane architecture around (effectively pull-only) Prometheus.

What are the benefits or main reason for replacing zabbix with prometheus, especially since zabbix introduced prometheus checks?

Re: Systems Monitoring with Prometheus and Grafana

#75

Post by unixhero »

halfmatthalfcat wrote: post #50
AlphaSite wrote: post #45

Earlier quoted context omitted.

Doesn’t it also have cardinality issues?

You get high enough (hitting a metric with >100k unique labels), queries become unmanageable and incredibly slow when backed by the stock datastore (tsdb). However there are backing datastores (TimescaleDB, InfluxDB, VictoriaMetrics, etc) that ingest Prometheus metrics and enable higher cardinality.

This is where I fall off.

Is Prometheus a DB that (can) forward data to another DB?

Re: Systems Monitoring with Prometheus and Grafana

#76

Post by unixhero »

rollulus wrote: post #55
fnord123 wrote: post #43

Earlier quoted context omitted.

>Promethii Prometheuses. ii is for latin words. Prometheus is/was Greek. I guess you could use Prometheoí but it would quickly derail any conversation. :)

There was a talk on Promcon 2016 about this subject [1]. The conclusion was: in English, indeed, Prometheuses. In Ancient Greek: Prometheis. [1]: https://www.youtube.com/watch?v=B_CDeYrqxjQ

I kind of like the Ancient Greek version.

Re: Systems Monitoring with Prometheus and Grafana

#77

Post by Jedd »

goranerik wrote: post #74
Jedd wrote: post #26

We've got a somewhat similar landscape, on a pretty sizeable network - big investment in Zabbix and looking to move, perhaps slowly and perhaps only in part, towards Prometheus. Coming from a monitoring system that supports push and pull with elegant auto-discovery, we're struggling to work out a sane architecture around (effectively pull-only) Prometheus.

What are the benefits or main reason for replacing zabbix with prometheus, especially since zabbix introduced prometheus checks?

Good and valid question.

I expect we won't outright replace, but rather augment, especially in spaces where a host-centric tool like Zabbix isn't ideal.

Partly it's driven by a need to monitor things like k8s (in the form of openshift) and pub/sub systems (eg kafka), and to integrate with other data sources (eg elastic).

Possibly more compelling is the need to do more sophisticated things with our data than we can conveniently accomplish with the Zabbix data store -- it's not the DB performance or scalability (PostgreSQL and optionally TimescaleDB) so much as dealing with the schema. Mildly sophisticated wrangling of our data ranges from difficult to impossible.

There's a couple of ways around that - bespoke tooling to facilitate ad hoc interrogations into the DB, duplicate the data at ingest time into multiple datastores, frequent ETL of the Zabbix SQL data into long term (time series) storage. None of these are great options. Plus we're fans of Grafana, so some of our decisions are, and will be, based around maintaining or improving end-user experience of that tool -- and while the Zabbix integration is excellent, the Prometheus integration is even better, so (on the end-user side) that's a highly compelling path.

Re: Systems Monitoring with Prometheus and Grafana

#78

Post by halfmatthalfcat »

unixhero wrote: post #75
halfmatthalfcat wrote: post #50

Earlier quoted context omitted.

You get high enough (hitting a metric with >100k unique labels), queries become unmanageable and incredibly slow when backed by the stock datastore (tsdb). However there are backing datastores (TimescaleDB, InfluxDB, VictoriaMetrics, etc) that ingest Prometheus metrics and enable higher cardinality.

This is where I fall off. Is Prometheus a DB that (can) forward data to another DB?

Prometheus is a data format but it's also a "suite" of tools on top of that data format.

Usually what happens is your app, db, whatever will expose metrics (http request status, average response time, etc) in the Prometheus format which is then scraped by the Prometheus ingestor. The ingestor stores those metrics in a (short-term) datastore called TSDB. Prometheus also ships with a little web UI as well that can query those metrics in TSDB.

However Prometheus allows scraping from the ingestor (usually federation) or pushing into an external datastore that is usually more performant than TSDB.

So when people say "Prometheus", they usually are talking about the suite of tools however practically it's really just the format of the metrics data.

Re: Systems Monitoring with Prometheus and Grafana

#79

Post by sagichmal »

jldugger wrote: post #39
sagichmal wrote: post #19

Earlier quoted context omitted.

High fidelity operational metrics have a useful half life measured in days or weeks. Read patterns for longer term use cases are also categorically different. Best architecture is to have a separate system for long term stuff, which treats Prometheus as a data source. Then Prometheus can drop after 14-28d.

> High fidelity operational metrics have a useful half life measured in days or weeks. Depends on the metric IMO. There's a ton of use you can get out of forecasting and seasonality for anomaly detection, but you need data going back for that to have any chance. Many relevant operations metrics exhibit three levels of seasonality: daily (day/night) weekly (weekday/weekend) and annual (holidays, superbowls, media even…

I agree that it's important to keep some telemetry data for the long term. My point is that you need fewer and less granular metrics for those use cases, and that the access patterns are sufficiently different from real-time operations, that they're most effectively served by two completely different systems.

Re: Systems Monitoring with Prometheus and Grafana

#80

Post by VectorLock »

skohan wrote: post #68
dijit wrote: post #14

Grafana truly is best in class, but I have strong reservations about Prometheus. I really want to like it, it’s just so _easy_, publish a little webpage with your metrics and Prometheus takes care of the rest. Lovely. But I often find that the cardinality of the data is substantially lower than even the defaults of alternatives (influxdb has 1s and even Zabbix has 5s). Not to mention the lost writes (missing data poi…

> Grafana truly is best in class Really? Recently we've been playing with Chronograf with InfluxDB and most people find it a lot nicer to work with than Grafana (specifically because it makes discoverability a lot nicer)

Recent Grafana's Explore interface is much nicer.
Post reply on HN