Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best monitoring system?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 128 posts

Re: Ask HN: Best monitoring system?

#91

Post by jrv »

mspaulding06 wrote: post #61
solipsism wrote: post #58

Earlier quoted context omitted.

It should be noted that Riemann is different from Nagios and Prometheus in that it's push-based. The other two are primarily pull-based.

Yep, I think that is Riemann's greatest advantage. I don't like the fact that with Prometheus I have to set up some sort of service discovery mechanism. Right now we're using Foreman to manage our clusters, so most likely I would have to use the file service discovery type and query the Foreman API to produce a YAML or JSON file to provide a list of static hosts.

If your monitoring system doesn't have knowledge of what's supposed to be out there (like via service discovery), how do you know when something is broken/down/missing?

Re: Ask HN: Best monitoring system?

#92

Post by jrv »

mspaulding06 wrote: post #38
sagichmal wrote: post #19

Prometheus is absolutely the way you should be going. All of the other systems I'm seeing mentioned here — Nagios, Icinga, check_mk, Zabbix, Sensu — are host-centric and are very awkward when you try to bend them to fit modern (containerized, etc.) workloads.

Yeah, containerization is a good point. We've been starting to deploy apps on Docker within the last 6 months. What does Prometheus offer that makes it better for containerization?

Mostly, it has great support for service discovery for many major cloud and container platforms (Kubernetes, Marathon, EC2, Azure, Zookeeper, Consul, ...). So it can not only go out and pull metrics data directly from instances as they float around your dynamic cluster scheduler, but also attach identity metadata (as provided by the service discovery) to the time series collected from each instance. For example, you may map EC2 tags or Kubernetes labels into your Prometheus time series labels to give you more useful metrics. There's also a way to plug in your own custom service discovery. Also, Kubernetes exports native Prometheus metrics since quite a while already.

Borg inspired Kubernetes. Borgmon inspired Prometheus. So naturally it works well together with a dynamically scheduled world.

Re: Ask HN: Best monitoring system?

#93

Post by jrv »

falcolas wrote: post #29
sagichmal wrote: post #22

Earlier quoted context omitted.

Active polling scales _better_ than the alternative. You directly manage the ingest rate at the server side, rather than DDOSing your monitoring infrastructure when you scale out your services to deal with load.

> You directly manage the ingest rate at the server side Why in the world would you want to manage (which I'm reading as throttle) the ingest rate of a monitoring service? That strikes me as a recipe for missing important events. Monitoring should be a fairly stable load rate. If you're setting up monitoring, and your machine can't handle the load of all the data points coming in, you need to shard out, not drop data…

The pulling itself has never been a scaling/bottleneck concern in Prometheus, especially since no individual events are transferred, but just the current state of each metric. The bottleneck is usually the storage's sustained ingestion speed (disk IO or similar).

A single big Prometheus server can easily do millions of time series, and in an older record, 800,000 samples stored per second. You could monitor e.g. 10,000 hosts with that with quite some detail, and the bottleneck is still not the pull aspect.

Re: Ask HN: Best monitoring system?

#94

Post by bbrazil »

falcolas wrote: post #5

Just my opinion, but I won't use Prometheus, because of the active polling model. It won't scale without a number of workarounds. My preferred method is Icinga2 (a Nagios clone with better configuration and clustering built-in) with reports coming in via passive NSCA. Toss in Graphite (or I'm warming up to Grafana on Influx) with some ability to write alerts against those reported metrics, and you're as close to idea…

> but I won't use Prometheus, because of the active polling model.

> [Grafana+Ichinga] they're so rock solid (and scale like mad) I have a hard time not recommending them.

As a Prometheus developer I have seen a significant number of users who moved from Graphite because they found it doesn't scale and was far from rock solid for them, requiring regular manual care and feeding. By contrast Prometheus seems to be working pretty well for them at what we would consider to be a moderate load. I've heard similar about Nagios/Ichinga.

Push vs. pull is largely not relevant for scaling (pull is slightly better in this regard, but only slightly). I've been involved with some extremely large scale monitoring systems, and the fact they were pull was never relevant to scaling them.

May I ask what you consider to be a high level of scale?

> It won't scale without a number of workarounds.

There are very very few systems who won't scale without workarounds. That's the nature of scaling a non-trivial system.

Re: Ask HN: Best monitoring system?

#95

Post by jrv »

moehm wrote: post #80

Most people here are recommending Prometheus. What is the best monitoring system to monitor good old infrastructure software like DNS servers, IMAP/SMTP server etc? Is Prometheus a reasonable choice for those as well?

Yes, Prometheus is a great choice for that as well. It's pretty easy to write integrations (we call them exporters) to get metrics out of existing third-party systems that you cannot easily instrument directly. Here's a list of exporters we already know about, but it's usually easy to write one of your own if it doesn't exist yet: https://prometheus.io/docs/instrumenting/exporters/

Re: Ask HN: Best monitoring system?

#96

Post by bbrazil »

mspaulding06 wrote: post #59
acd wrote: post #37

Prometheus.io which is a modern fresh monitoring system that I would checkout if replacing a legacy system. Also take a look at Riemann which is system monitoring written in Clojure. Riemann should be good for monitoring latency of the system. If it helps here is Slidedeck from Spotify how they do their monitoring https://www.netways.de/fileadmin/images/Events_Trainings/Eve...

I like Riemann and actually read The Art of Monitoring ( https://artofmonitoring.com/ ) which was a great book. There are two big downsides for me on this one. First, you MUST build your monitoring solution from scratch and you MUST learn Clojure (which can be hard to get a whole team to agree to). Second, there's no alerting dashboard, which makes it difficult to see the overall state of the clusters you're dealing…

> you MUST learn Clojure (which can be hard to get a whole team to agree to)

For any sufficiently advanced monitoring system, you're going to have to learn some form of DSL/language to take advantage of it.

I wouldn't consider this to be a major issue, more part of the irreducible complexity of the problem. The Riemann examples I've seen all seemed pretty readable, and routing alerts is not world away from what you'd be doing in say a Prometheus Alertmanager config; just with S-Expressions against YAML.

Re: Ask HN: Best monitoring system?

#97

Post by tayo42 »

caw wrote: post #62
tayo42 wrote: post #40

Earlier quoted context omitted.

Datadog was down so often when I had to use it. It felt so unreliable. We used to monitor hosts and it got to the point where checking if datadog was down was part of troubleshooting..

How long ago was that? We just switched in January and it's been pretty reliable. As far as Datadog goes, it's the most team friendly dashboard system we've used. We had a specialty monitoring system for one application stack previously, and no one made custom dashboards there or even just looked at the data. Now we've got custom dashboards out the nose and we're gradually consolidating to a "best of" dashboard for e…

The ui is pretty nice for sure, especially compared to some other things im using.

I used it up until two months ago when I left that job. There was 2000~ servers monitored I think.

Re: Ask HN: Best monitoring system?

#98

Post by mspaulding06 »

jrv wrote: post #91
mspaulding06 wrote: post #61

Earlier quoted context omitted.

Yep, I think that is Riemann's greatest advantage. I don't like the fact that with Prometheus I have to set up some sort of service discovery mechanism. Right now we're using Foreman to manage our clusters, so most likely I would have to use the file service discovery type and query the Foreman API to produce a YAML or JSON file to provide a list of static hosts.

If your monitoring system doesn't have knowledge of what's supposed to be out there (like via service discovery), how do you know when something is broken/down/missing?

Riemann injects expire events into the event stream when it doesn't hear from a service for a while. Each event has a TTL that it uses to figure this out. Then of course you can alert on the expired events or take other actions.

Re: Ask HN: Best monitoring system?

#99

Post by bbrazil »

mspaulding06 wrote: post #98
jrv wrote: post #91

Earlier quoted context omitted.

If your monitoring system doesn't have knowledge of what's supposed to be out there (like via service discovery), how do you know when something is broken/down/missing?

Riemann injects expire events into the event stream when it doesn't hear from a service for a while. Each event has a TTL that it uses to figure this out. Then of course you can alert on the expired events or take other actions.

How does this tell you that something is missing if it never reported, or that something is no longer meant to exist?

Re: Ask HN: Best monitoring system?

#100

Post by mspaulding06 »

bbrazil wrote: post #99
mspaulding06 wrote: post #98

Earlier quoted context omitted.

Riemann injects expire events into the event stream when it doesn't hear from a service for a while. Each event has a TTL that it uses to figure this out. Then of course you can alert on the expired events or take other actions.

How does this tell you that something is missing if it never reported, or that something is no longer meant to exist?

It's true, you can't tell if something is missing if it never reported. I guess it depends on your requirements. In a situation where your workloads are ephemeral you may not care as much if all of your services have reported as long as most have. In the case that something should no longer exist you could write that functionality into your Riemann configuration. As an example, I could write a "dead service" stream processor that is used to inform Riemann that a service should no longer exist. When a "dead service" event is injected into the event stream for a particular service, Riemann will note the dead service and ignore future events from it.
Post reply on HN