Show HN: Monitoror – Unified monitoring wallboard

I think the fundamental question of all such tools is "Why are we watching this, and what are we looking for," and there are limited but nonzero good reasons to have a display. "Someone should look at open PRs if there are too many" is a bad one - the number doesn't tell you about the urgency of the existing PRs. If you want to respond promptly, respond to all of them promptly.

"We need to know if we're falling behind" is a possible reason to create an alert, not a dashboard. If you really want people to drop what they're doing and triage issues if there are too many, make an alert. If you don't, you'll just get a rectangle that turns red at some point and train people to ignore red rectangles on the board. (Relatedly: I added a pageable alert to my team a few years back to check whether there are a large number of non-pageable alerts, because it usually means something has gone wrong at a low level and we should investigate urgently. It's worked out pretty well, but the alert looks only at tickets created by our monitoring systems, not at tickets created by humans.)

"We need to see if we're getting worse" is a reason to have managers review graphs periodically, not a reason for anyone to stare at a single display. You can't track long-term trends from a status board.

"I need to see what to work on" is a valid reason, but much more useful in the form of a website you can visit on your own computer with links to PRs, not a raw number on a TV screen. (My team has a TV showing open tickets in our queue, both support tickets and automated alert, but we all have an equivalent link locally, too. Showing the names of tickets is useful for "Hey teammate, can you look at the second ticket there? Sounds related to a thing you were working on.")

I'd say there are roughly two useful cases for screens like this. One is to show to internal customers, so they say "oh, service X is yellow, so the slowness I"m seeing isn't just me, I'll do something else for a while." But those screens aren't primarily for the team that owns the product, they're for teams that depend on the product. (Such status boards can be either automated or manual.) The other is to show graphs of various metrics to see abnormal behavior, with the idea that no action is ever triggered by someone looking at the graph, but if you're already investigating something, it's useful to say "Hey, that's funny, this other thing spiked at about the same time even though it's within acceptable limits" and then you have a clue for investigation.

https://demo.monitoror.com/?configUrl=https://monitoror.com/... is full of things that aren't visualizations at all (no graphs, no sense of whether things are abnormal but not past an alerting threshold, etc.) and are in fact alerts (the website is fine, one PR failed, the QA nodes are ... doing something but there isn't enough space to see what is wrong).

If you want some graphs, great. If you want your team to look up every few minutes and poll some graphs (or worse, some colored rectangles) to figure out what they're supposed to be doing, consider that polling is usually the wrong approach.

(To be clear, this is a criticism of the choice of demo data, not of the product overall. A product like this has its uses, but "our alerting system is people looking up at the TV" is not one of them.)

So how many times was an issue missed because you weren't in the office, or because you were looking at your own screen and not dashboards at the moment?

Humans are incredibly powerful, but our whole job as SREs is to make things reliable, repeatable, and scalable. We're doing an industry-wide migration from elegantly hand-crafted LAMP stacks running SSH to Kubernetes and infrastructure-as-code, not because you can't fix problems with SSH (you can, and you can usually fix them faster and better) but because you can't scalably fix problems with SSH. Similarly, if a human found an issue and alert didn't trip, I'd count that as a bug/missing feature in the monitoring.

It's valuable while you're still small and working out your monitoring to keep a human in the loop - but at some point you need to get rid of that single point of failure. By all means, rely on a human to figure out where your alerting is lacking (just like you rely on a human to write the infrastructure-as-code), but you should eventually not rely on human intervention to actually keep incidents from happening.

It's MIT licenced by the way.