Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Systems Monitoring with Prometheus and Grafana

flightaware.engineering

Post reply on HN
Print view View original

41–50 of 107 posts

Re: Systems Monitoring with Prometheus and Grafana

#41

Post by jeffbee »

cmckn wrote: post #3

Prometheus is great. I first heard about it at KubeCon last fall, and kind of shrugged it off as one of those fledgling "cloud native" projects that I probably didn't need or didn't have time to learn. There's actually a lot of adoption, you can find great exporters and grafana dashboards for almost any OSS you're running today. I started collecting metrics from Zookeeper and HBase in about an hour, having never had…

Interesting to say that Prometheus is "fledgling". The project is almost 7 years old and the Google thing on which it is based is ~15 years old.

Re: Systems Monitoring with Prometheus and Grafana

#43

Post by fnord123 »

EdwardDiego wrote: post #23
dijit wrote: post #14

Grafana truly is best in class, but I have strong reservations about Prometheus. I really want to like it, it’s just so _easy_, publish a little webpage with your metrics and Prometheus takes care of the rest. Lovely. But I often find that the cardinality of the data is substantially lower than even the defaults of alternatives (influxdb has 1s and even Zabbix has 5s). Not to mention the lost writes (missing data poi…

We've used Thanos to aggregate multiple Prometheus (Promethii?) across our clusters to enable us to scale, each Prometheus deals only with a subset of scrape targets. Biggest issue I've had was an app that was accidentally publishing several thousand metrics which caused the default scrape timeout of 15s to kick in. (It was publishing Kafka lag per consumer group per topic, which was fine and dandy, until someone rel…

>Promethii

Prometheuses.

ii is for latin words. Prometheus is/was Greek. I guess you could use Prometheoí but it would quickly derail any conversation. :)

Re: Systems Monitoring with Prometheus and Grafana

#44

Post by molecule »

bacondude3 wrote: post #34

What's a good alternative to Prometheus when pulling stats is impractical? Say I want to monitor a personal laptop like I would a server. It will change networks and IP addresses, so pulling would be impractical to configure, whereas the laptop could easily(?) push its stats to a remote server.

Telegraf + InfluxDB?

Re: Systems Monitoring with Prometheus and Grafana

#45

Post by AlphaSite »

ecnahc515 wrote: post #24
dijit wrote: post #14

Grafana truly is best in class, but I have strong reservations about Prometheus. I really want to like it, it’s just so _easy_, publish a little webpage with your metrics and Prometheus takes care of the rest. Lovely. But I often find that the cardinality of the data is substantially lower than even the defaults of alternatives (influxdb has 1s and even Zabbix has 5s). Not to mention the lost writes (missing data poi…

> Erything about the design (polling, single database) tells me that it was designed primarily to sit alongside something small. Prometheus is designed to be "functionally sharded". You shouldn't be running one "mega prometheus". Often it's something like 1 Prometheus per-team, depending on the amount of metrics each produces. You can use federation at lower resolutions or a one of the distributed setups (Thanos/Cort…

Doesn’t it also have cardinality issues?

Re: Systems Monitoring with Prometheus and Grafana

#46

Post by abhishekjha »

molecule wrote: post #44
bacondude3 wrote: post #34

What's a good alternative to Prometheus when pulling stats is impractical? Say I want to monitor a personal laptop like I would a server. It will change networks and IP addresses, so pulling would be impractical to configure, whereas the laptop could easily(?) push its stats to a remote server.

Telegraf + InfluxDB?

This is my setup on all my raspberrypis. I have not be been able to figure out how to monitor a cluster though. I saw that grafana free tier doesn’t allow a cluster of servers getting monitored. I have telegraf + influxdb + grafana installed on all my servers.

Re: Systems Monitoring with Prometheus and Grafana

#47

Post by abhishekjha »

edoceo wrote: post #5
cmckn wrote: post #3

Prometheus is great. I first heard about it at KubeCon last fall, and kind of shrugged it off as one of those fledgling "cloud native" projects that I probably didn't need or didn't have time to learn. There's actually a lot of adoption, you can find great exporters and grafana dashboards for almost any OSS you're running today. I started collecting metrics from Zookeeper and HBase in about an hour, having never had…

My Prometheus system is a $10/mo Linode. It collects from 27 other hosts, and at least 100 services distributed across those hosts - doesn't even break a sweat. All the exporters run through a wireguard VPN. Prometheus is great for a small/medium SaaS type environment.

What do you use as a frontend? As far as I could tell grafana free tier doesn’t allow monitoring cluster of servers.

Re: Systems Monitoring with Prometheus and Grafana

#48

Post by RhodesianHunter »

AlphaSite wrote: post #45
ecnahc515 wrote: post #24

Earlier quoted context omitted.

> Erything about the design (polling, single database) tells me that it was designed primarily to sit alongside something small. Prometheus is designed to be "functionally sharded". You shouldn't be running one "mega prometheus". Often it's something like 1 Prometheus per-team, depending on the amount of metrics each produces. You can use federation at lower resolutions or a one of the distributed setups (Thanos/Cort…

Doesn’t it also have cardinality issues?

There's no real way to develop a metrics system without cardinality issues. Where you draw the line depends on the backing database but they're all fairly constrained.

Re: Systems Monitoring with Prometheus and Grafana

#49

Post by RhodesianHunter »

kasey_junk wrote: post #13

I have a love/hate relationship with Prometheus. If I had no budget for metrics its likely the thing I would reach for, but I’m dying for someone to open source a ‘next level’ metrics system (something like Monarch or Circonus but free). But woe betide the team that has to run it as a service. Not that other metrics systems are better but Prometheus can be brutal in that space. As a ‘squad level’ tool its really good…

>I’m dying for someone to open source a ‘next level’ metrics system

For the time being this is a "full team working full 40 hour weeks for year(s)" problem, so I'd be shocked to see it done open source.

Re: Systems Monitoring with Prometheus and Grafana

#50

Post by halfmatthalfcat »

AlphaSite wrote: post #45
ecnahc515 wrote: post #24

Earlier quoted context omitted.

> Erything about the design (polling, single database) tells me that it was designed primarily to sit alongside something small. Prometheus is designed to be "functionally sharded". You shouldn't be running one "mega prometheus". Often it's something like 1 Prometheus per-team, depending on the amount of metrics each produces. You can use federation at lower resolutions or a one of the distributed setups (Thanos/Cort…

Doesn’t it also have cardinality issues?

You get high enough (hitting a metric with >100k unique labels), queries become unmanageable and incredibly slow when backed by the stock datastore (tsdb). However there are backing datastores (TimescaleDB, InfluxDB, VictoriaMetrics, etc) that ingest Prometheus metrics and enable higher cardinality.
Post reply on HN