31–40 of 61 posts
I also use `failure-monitor` which is Python service that monitors `journald`.
Files on Github for those interested:
I was building an elaborate job monitoring system, but then I realized that what I really need is monitoring the actual end to end functionality. For example, instead of monitoring my Minecraft server process that OpenRC spawns, I have a dedicated monitoring server that actually queries the server for version, number of players, etc. Same for websites, etc. Think of it as periodically running an integration test on a…
I'm of the opinion that having charts and graphs to rely on can focus troubleshooting resources more quickly onto the most actionable areas.
I don't monitor services at that level at all, because it means basically nothing. More acutely: the the lack of a notification doesn't tell mean everything is "ok". I tend to monitor the actual service. If it's a web server, have something checking that a specific URL is working (tip: use something specific, not /). Likewise any other network service is pretty easy to monitor. For backups, check the date on the most…
Checks from the point of view of an end user are the gold standard if the service is functioning and functioning well enough. I very much agree with this. For example, with the case of postgres, something like sharp increases or decreases in query throughput or query durations is something to alert on, because this will negatively impact the applications depending on it.
However, we have incrementally implemented additional checks and dependencies between checks to speed up troubleshooting complex systems during an emergency. Instead of on-call having to, e.g., check postgres, check patroni, check consul, check consul server cluster, go back, check network, check certificates... zabbix can already compile this into a statement like "postgres is down, but that is caused by patroni not reaching the DCS, but that's caused by the consul client being down.. however, the service is running and the certificates are fine and the consul-server cluster is also fine".
Short answer: Prometheus + Grafana + Alertmanager. prometheus_node_exporter has an option to export SystemD service status and you can alert on failed services, and you can use Alertmanager to configure multiple types of alarms, including repeats so you don't forget. Long answer: Whenever I've started to add alerting and monitoring to a system, I end up wanting to add more things each time, so I find it valuable to s…
> Short answer: Prometheus + Grafana + Alertmanager. Or, a higher-level recommendation, appropriate for most SMBs: sign up for Grafana Cloud's managed prometheus+grafana (or any equivalent external managed monitoring stack), and then follow their setup instructions to install their grafana-agent monitoring agent package (which sticks together node_exporter, several other optional exporters enable-able with config sta…
I must have spent about a week trying to learn just enough about prometheus and grafana (I had used grafana before with influx but for a different purpose) so that we could monitor temperature, memory, cpu, and disk (the bare minimum).
The goal was to have a single dashboard showing these critical metrics for all servers (No luck. After a week I had nothing to show for.
So I turned to Netdata. A one liner on each server and we had super sexy and fast dashboard for each server. No birds eye view, but fine. I then spent maybe 3-4 days trying to figure out how to get alerting to work (just email, but fine) and get temperature readings (or something like that).
No luck. By the end of week 2 I still had nothing, but a bunch of servers shutting down during peak hours.
Week 3 I said fuck it I'll do the stupidest thing and write my own stack. A bunch of shell scripts, deployed via ansible, capturing any metric I could think of, managed by systemd, posting to a $5/month server running a single nodejs service that would do in memory (only) averages, medians etc, and trigger alerts (email, sms, Slack maybe soon) when things get yellow or red.
By week 4 we had monitoring for all servers and for any metric we really needed.
Super cheap, super stable and absolutely no maintenance required. Sure, we probably can't monitor hundreds of servers or thousands of metrics, but we don't need to.
I really wanted to use something else, but I just couldn't :(
I use Nagios, easy, lean and gets the job done
Things get deployed by the automatic deployment system. If they go in cron, they are supervised by a program called errorwatch which does all the things that you want in a one-shot supervisor: logging, error codes, time bounds, checking for right output, checking for wrong output. If they are daemonic, they get /etc/init.d/ start/stop scripts that have been tested.
If they have a habit of dying and we can't afford that and we can't fix it, we run them from daemontools instead of init.d.
> "I have a backup job that is triggered by a timer. I want to know when that job fails so I can investigate and fix it." This is really more in the realm of a shell script. You could do this verbosely: #!/bin/sh /path/to/my/backup_job if [ $? -ne 0 ] then /path/to/my/failure_alert fi ...or, you could do this tersely: #!/bin/sh /path/to/my/backup_job || /path/to/my/failure_alert The wrapper script would go into your…
Why does email feel wrong? I find it a pretty viable solution.
If you are ok with a Saas and if it's just scheduled jobs that you are monitoring, there are a number of monitoring tools where you tell when job completes (with a http request) and a missing ping (after a grace period) means that it failed. I think https://deadmanssnitch.com/ may have been the original service for this. https://healthchecks.io/ has a fairly generous free tier that I use now. There are others that do…