Ask HN: Best monitoring system?

It depends on your architecture and scale. There is no "best", just "best we've found for this" and "best given other constraints".

This is yet another point where DevOps is not "devs doing ops" but "operations building and deploying with all the tools of modern software development". You need a subject matter expert.

What are you monitoring? Do you care about availability or performance or both? Scale? Do you have services or servers? Do you manage the underlying hardware? Do you need to track which hardware boxes have which VMs or containers?

There are a million questions to answer. One big set of them: what do you dislike about Nagios? Make sure that you don't get those problems with the next one, but also make sure you get something that does what you need as well as what you want.

Yes, check_mk is great. I rolled out a 30000+ checks installation on a single server today. I've looked at most monitoring products on the market and check_mk easily wins. It's really popular in Germany, many huge companies are using it.

The English documentation isn't that great, the German one is better. That being said, it's mostly self-explanatory and all checks are very well documented in man pages.

My favorite features:

- Auto discovery for literally everything, including SNMP interfaces.

- Fine grained rule system for customizing check threshold and parameters

- All configuration is automatically versioned and you can integrate it with Git - this includes the changes you make in the web interface.

- It's very easy to set up a distributed monitoring system (multisite) with a central node which aggregates all states and replicated configuration changes, yet each site is fully autonomous.

- The agent takes zero network input, so no attack surface.

- Even though it's extremely featureful, it's architecture is very simple and it's easy to contribute code and write custom checks.

Their Git is public: http://git.mathias-kettner.de/git/?p=check_mk.git;a=shortlog...

It works well with Naemon and Nagios 4. Been using it for a number of projects, ask me anything!

Monitoring systems not to use:

  - Shinken (zero security awareness and dishonest PR, when I tried it, it has so many bugs that I wouldn't ever trust it to keep my data safe)

  - Zabbix (it has some brilliant features, but the architecture is a mess, configuration and time series stored in a MySQL database which is hard to manage and automate, I found it cumbersome to debug, written in PHP)

Thanks for the comprehensive reply. Briefly I would say that we care about availability more than performance, though both are important. We're running somewhere on the order of 2000 VMs w/ ESX with some bare metal systems running database clusters. We have a separate team that manages the hardware infrastructure and they have their own monitoring and alerting system. I'm mostly concerned about preventing downtime for the application cluster, alerting the right people via the right means (chat, email, pagerduty) when something does go down, and getting some high resolution graphs for analysis.

Give opsee a try https://opsee.com/

> You directly manage the ingest rate at the server side

Why in the world would you want to manage (which I'm reading as throttle) the ingest rate of a monitoring service? That strikes me as a recipe for missing important events.

Monitoring should be a fairly stable load rate. If you're setting up monitoring, and your machine can't handle the load of all the data points coming in, you need to shard out, not drop data points.

> Active polling scales _better_ than the alternative

Exceptional claims require exceptional evidence. Personally, I have never heard of anything that is actively pinging outside services performing better than receiving and processing data passively.

Prometheus will scale better than Nagios running active checks since it won't be using subprocesses, but it is still going to require more overhead than a service receiving passive reports.