Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Systems Monitoring with Prometheus and Grafana

flightaware.engineering

Post reply on HN
Print view View original

81–90 of 107 posts

Re: Systems Monitoring with Prometheus and Grafana

#81

Post by abhishekjha »

michaelmcdonald wrote: post #73
abhishekjha wrote: post #46

Earlier quoted context omitted.

This is my setup on all my raspberrypis. I have not be been able to figure out how to monitor a cluster though. I saw that grafana free tier doesn’t allow a cluster of servers getting monitored. I have telegraf + influxdb + grafana installed on all my servers.

Could you expand on what you mean by: > grafana free tier doesn’t allow a cluster of servers getting monitored. Is there a particular aspect of the cluster you're missing? Is it that you don't want individual server metrics?

I have telegraf + influxdb + grafana-server installed on each of my Rpis giving me multiple dashboards. I want only one grafana-server dashboard where all the telegraf metrics could be seen.

Re: Systems Monitoring with Prometheus and Grafana

#82

Post by abhishekjha »

dewey wrote: post #72
abhishekjha wrote: post #47

Earlier quoted context omitted.

What do you use as a frontend? As far as I could tell grafana free tier doesn’t allow monitoring cluster of servers.

You could self host it.

Can I self host for monitoring cluster of servers? Currently I have grafana installed on each of my servers and I am having to monitor them individually. I want a centralised dashboard over telegraf + influxdb.

Re: Systems Monitoring with Prometheus and Grafana

#83

Post by detaro »

abhishekjha wrote: post #82
dewey wrote: post #72

Earlier quoted context omitted.

You could self host it.

Can I self host for monitoring cluster of servers? Currently I have grafana installed on each of my servers and I am having to monitor them individually. I want a centralised dashboard over telegraf + influxdb.

Why would you install Grafana + Influx on each server instead of one central one?

Re: Systems Monitoring with Prometheus and Grafana

#84

Post by abhishekjha »

detaro wrote: post #83
abhishekjha wrote: post #82

Earlier quoted context omitted.

Can I self host for monitoring cluster of servers? Currently I have grafana installed on each of my servers and I am having to monitor them individually. I want a centralised dashboard over telegraf + influxdb.

Why would you install Grafana + Influx on each server instead of one central one?

I haven't spent much time on this but most of the docs were for setting it up on each hosts. Is there a proper tutorial for clusters?

Also I wanted to keep the monitoring unaffected for other servers if one of them go down. If I setup a central server for monitoring then that becomes a single point of failure.

Re: Systems Monitoring with Prometheus and Grafana

#85

Post by djmetzle »

I thought this article, while a little dry, was very illuminating. It sounds Hyperfeed is running at the very least "Medium Data" (we all thing our Data is Big!). And i think it is fascinating to hear of a case where Prometheus is plainly a bad fit for it's intend purpose. It sounds like cardinality explosion around their ML models was a really bad fit for Prometheus. Its great to hear about deployments "in-situ", and people appreciating where it works well, and where it doesn't.

Re: Systems Monitoring with Prometheus and Grafana

#86

Post by chinhodado »

akulkarni wrote: post #11
halfmatthalfcat wrote: post #2

Prometheus and Grafana are awesome, use them personally for all my monitoring. However I’m still trying to nail down my high cardinality/highly unique metrics-like data story. What are people using? I’ve heard a combination of Cassandra/BigTable and Spark as a potential solution?

TimescaleDB is a long-term storage option for Prometheus metrics, has no problem with high-cardinality, and now natively supports PromQL (in addition to SQL) [0] (Disclaimer: I work at Timescale) [0] https://github.com/timescale/timescale-prometheus

I'm just starting to look into this and have a question. If I can export my metrics directly to TimescaleDB and it supports visualization with Grafana, is there any reason to go through Prometheus?

Re: Systems Monitoring with Prometheus and Grafana

#87

Post by bacondude3 »

cassianoleal wrote: post #65
bacondude3 wrote: post #34

What's a good alternative to Prometheus when pulling stats is impractical? Say I want to monitor a personal laptop like I would a server. It will change networks and IP addresses, so pulling would be impractical to configure, whereas the laptop could easily(?) push its stats to a remote server.

I've been using ZeroTier [0] so whenever my laptop is online, it's also accessible by Prometheus so it can be scraped. I run ZeroTier in a container on my router, and on each "road warrior" that needs LAN access (laptop, phone...). The caveat is that I have no metrics when the laptop is offline but that doesn't happen very often anyway. [0] https://www.zerotier.com/

That looks like a really cool solution. Thanks for the suggestion!

Re: Systems Monitoring with Prometheus and Grafana

#88

Post by bacondude3 »

MetalMatze wrote: post #63
bacondude3 wrote: post #34

What's a good alternative to Prometheus when pulling stats is impractical? Say I want to monitor a personal laptop like I would a server. It will change networks and IP addresses, so pulling would be impractical to configure, whereas the laptop could easily(?) push its stats to a remote server.

Prometheus supports writing (replicating) data to a remote endpoint on a per scrape basis with a protocol called remote-write. You can pretty easily set that up on any Prometheus instance. There are quite some implementations to receive those remote-write requests: https://prometheus.io/docs/operating/integrations/#remote-en... You're probably exactly looking for something like that. In fact, I've given a talk about…

Oh, I had no idea about that. Thanks!

Re: Systems Monitoring with Prometheus and Grafana

#89

Post by scaryclam »

ashtonkem wrote: post #57
dijit wrote: post #14

Grafana truly is best in class, but I have strong reservations about Prometheus. I really want to like it, it’s just so _easy_, publish a little webpage with your metrics and Prometheus takes care of the rest. Lovely. But I often find that the cardinality of the data is substantially lower than even the defaults of alternatives (influxdb has 1s and even Zabbix has 5s). Not to mention the lost writes (missing data poi…

My understanding is that Prometheus is designed for you to deploy multiple instances within your company, rather than deploying a limited number of instances for the company or division. So I would reasonably run a Prometheus instance by myself or with my neighboring teams rather than depending on a centralized instance run by $OPS.

This is how we use it and it works well. Other teams are also free to use whatever else they want and if we need an "overview" it's pretty easy to upstream certain metrics elsewhere (say, a centralised system run by ops) to collate together.

Being able to also control which metrics are important to my team vs the wider team is a BIG bonus of this sort of decentralised system.

Re: Systems Monitoring with Prometheus and Grafana

#90

Post by valyala »

kasey_junk wrote: post #13

I have a love/hate relationship with Prometheus. If I had no budget for metrics its likely the thing I would reach for, but I’m dying for someone to open source a ‘next level’ metrics system (something like Monarch or Circonus but free). But woe betide the team that has to run it as a service. Not that other metrics systems are better but Prometheus can be brutal in that space. As a ‘squad level’ tool its really good…

Could you give more context on Monarch or Circonus features that are missing in Prometheus?

BTW, I'm working on VictoriaMetrics - open source monitoring solution that works out of the box. See https://github.com/VictoriaMetrics/VictoriaMetrics

Post reply on HN