Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Systems Monitoring with Prometheus and Grafana

flightaware.engineering

Post reply on HN
Print view View original

31–40 of 107 posts

Re: Systems Monitoring with Prometheus and Grafana

#31

Post by Legogris »

dijit wrote: post #14

Grafana truly is best in class, but I have strong reservations about Prometheus. I really want to like it, it’s just so _easy_, publish a little webpage with your metrics and Prometheus takes care of the rest. Lovely. But I often find that the cardinality of the data is substantially lower than even the defaults of alternatives (influxdb has 1s and even Zabbix has 5s). Not to mention the lost writes (missing data poi…

Victoriametrics[0] is API-compatible with Prometheus but also a horizontally scalable, distributed and persisting timeseries database (cf influxdb). Together with vmagent it essentially becomes a HA drop-in replacement (almost) for Prometheus.

[0]: https://github.com/VictoriaMetrics/VictoriaMetrics

Re: Systems Monitoring with Prometheus and Grafana

#32

Post by Legogris »

halfmatthalfcat wrote: post #2

Prometheus and Grafana are awesome, use them personally for all my monitoring. However I’m still trying to nail down my high cardinality/highly unique metrics-like data story. What are people using? I’ve heard a combination of Cassandra/BigTable and Spark as a potential solution?

I'd be curious to hear if anyone has done serious evaluation of high-cardinality use-cases of Victoriametrics.

Re: Systems Monitoring with Prometheus and Grafana

#33

Post by Legogris »

site-packages1 wrote: post #12

What do you all do with the collected metrics over time? Do you store everything forever, drop everything after a couple weeks, or something on between? I've heard of people thinning out old data a bit (?) and storing it long term rather than storing everything. What's the usual thing people do?

One thing you can do is configure compressions, so essentially less recent data has lower time resolution and/or less cardinality. So some dimensions are dropped and you only have e.g. 1h resolution for data older than some threshold.

Re: Systems Monitoring with Prometheus and Grafana

#34

Post by bacondude3 »

What's a good alternative to Prometheus when pulling stats is impractical? Say I want to monitor a personal laptop like I would a server. It will change networks and IP addresses, so pulling would be impractical to configure, whereas the laptop could easily(?) push its stats to a remote server.

Re: Systems Monitoring with Prometheus and Grafana

#35

Post by cortesoft »

bacondude3 wrote: post #34

What's a good alternative to Prometheus when pulling stats is impractical? Say I want to monitor a personal laptop like I would a server. It will change networks and IP addresses, so pulling would be impractical to configure, whereas the laptop could easily(?) push its stats to a remote server.

Use a push gateway?

Re: Systems Monitoring with Prometheus and Grafana

#36

Post by bvrmn »

bacondude3 wrote: post #34

What's a good alternative to Prometheus when pulling stats is impractical? Say I want to monitor a personal laptop like I would a server. It will change networks and IP addresses, so pulling would be impractical to configure, whereas the laptop could easily(?) push its stats to a remote server.

You can use push approach. Like graphite. Grafana has excellent support of graphite queries. Also graphite has low-footprint implementation https://github.com/baverman/hisser

Re: Systems Monitoring with Prometheus and Grafana

#37

Post by Jedd »

EdwardDiego wrote: post #29
Jedd wrote: post #26

We've got a somewhat similar landscape, on a pretty sizeable network - big investment in Zabbix and looking to move, perhaps slowly and perhaps only in part, towards Prometheus. Coming from a monitoring system that supports push and pull with elegant auto-discovery, we're struggling to work out a sane architecture around (effectively pull-only) Prometheus.

There's a push gateway: https://github.com/prometheus/pushgateway

Yeah, I think we've looked at that. It provides push for the last mile, and I suppose you could wrangle some auto-discovery using that tooling, but you're still doing pull from Prometheus to that/those server(s).

We're still a bit stuck trying to replicate all the make-life-easy functionality we get with Zabbix sitting on a honking great PostgreSQL / Timescale database, with a bunch of proxies, and automated agent installs that auto-register.

There's places that doesn't work well (k8s, f.e.) but for conventional fleet metrics it's difficult to abandon.

Re: Systems Monitoring with Prometheus and Grafana

#38

Post by PerusingAround »

sagichmal wrote: post #20
latchkey wrote: post #15

Earlier quoted context omitted.

I found this interesting. My plan is to move from Prom to Victoria. https://medium.com/@valyala/measuring-vertical-scalability-f...

Woof, good luck. Not a great product.

Care to elaborate? At least a slight mention of why.

Re: Systems Monitoring with Prometheus and Grafana

#39

Post by jldugger »

sagichmal wrote: post #19
site-packages1 wrote: post #12

What do you all do with the collected metrics over time? Do you store everything forever, drop everything after a couple weeks, or something on between? I've heard of people thinning out old data a bit (?) and storing it long term rather than storing everything. What's the usual thing people do?

High fidelity operational metrics have a useful half life measured in days or weeks. Read patterns for longer term use cases are also categorically different. Best architecture is to have a separate system for long term stuff, which treats Prometheus as a data source. Then Prometheus can drop after 14-28d.

> High fidelity operational metrics have a useful half life measured in days or weeks.

Depends on the metric IMO. There's a ton of use you can get out of forecasting and seasonality for anomaly detection, but you need data going back for that to have any chance. Many relevant operations metrics exhibit three levels of seasonality: daily (day/night) weekly (weekday/weekend) and annual (holidays, superbowls, media events). Being able to forecast network traffic inbound on a switch to find problems would require you to have 1y of data, effectively. You _might_ be able to discard some of the data but you'd lose some of the predictive capacity for say, the Super Bowl.

Re: Systems Monitoring with Prometheus and Grafana

#40

Post by EdwardDiego »

Jedd wrote: post #37
EdwardDiego wrote: post #29

Earlier quoted context omitted.

There's a push gateway: https://github.com/prometheus/pushgateway

Yeah, I think we've looked at that. It provides push for the last mile, and I suppose you could wrangle some auto-discovery using that tooling, but you're still doing pull from Prometheus to that/those server(s). We're still a bit stuck trying to replicate all the make-life-easy functionality we get with Zabbix sitting on a honking great PostgreSQL / Timescale database, with a bunch of proxies, and automated agent in…

Yeah true, we find it easy for us because we're using K8s annotations for Prometheus scrape target discovery, so the gateway is just another target, and we're not running too many ephemeral jobs that we need more than one gateway.
Post reply on HN