Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best monitoring system?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 128 posts

Re: Ask HN: Best monitoring system?

#41

Post by bogomipz »

falcolas wrote: post #5

Just my opinion, but I won't use Prometheus, because of the active polling model. It won't scale without a number of workarounds. My preferred method is Icinga2 (a Nagios clone with better configuration and clustering built-in) with reports coming in via passive NSCA. Toss in Graphite (or I'm warming up to Grafana on Influx) with some ability to write alerts against those reported metrics, and you're as close to idea…

You are wrong about that. There is nothing inherent in a polling model that limits its ability to scale. You simply shard based on service or pod, or row(whatever your unit of scale is) and then you can always use consistent hashing/federation. Prometheus is a single binary. How many moving parts are involved in a Icinga 2 cluster?

Re: Ask HN: Best monitoring system?

#42

Post by williamstein »

Zenfinch wrote: post #10

If you can have a monitoring system in the cloud Datadog is a great choice. Good documentation, UI, many, many plugins and fair pricing (IMO). https://www.datadoghq.com/ (Im not affiliated with in any way other than using their product on a pet project with many moving parts).

Just make sure you understand their pricing (I didn't): http://sagemath.blogspot.com/2016/07/datadogs-pricing-dont-m...

Re: Ask HN: Best monitoring system?

#44

Post by bqe »

williamstein wrote: post #42
Zenfinch wrote: post #10

If you can have a monitoring system in the cloud Datadog is a great choice. Good documentation, UI, many, many plugins and fair pricing (IMO). https://www.datadoghq.com/ (Im not affiliated with in any way other than using their product on a pet project with many moving parts).

Just make sure you understand their pricing (I didn't): http://sagemath.blogspot.com/2016/07/datadogs-pricing-dont-m...

How could their pricing page be clearer? It says per host in fairly large letters underneath it.

I'm asking because I will be designing a similar page soon (that's also billed per host) and I'd like to avoid the same mistakes.

Re: Ask HN: Best monitoring system?

#45

Post by tedmiston »

Hynek Schlawack gave a talk at PyCon this year about using Prometheus and Grafana to unify monitoring metrics. Honestly the talk goes beyond my own understanding, but you may find it helpful. He's quite knowledgeable.

> To get real time insight into your running applications you need to instrument them and collect metrics: count events, measure times, expose numbers. Sadly this important aspect of development was a patchwork of half-integrated solutions for years. Prometheus changed that and this talk will walk you through instrumenting your apps and servers, building dashboards, and monitoring using metrics.

Abstract - https://us.pycon.org/2016/schedule/presentation/1601/

Slides - https://speakerdeck.com/hynek/get-instrumented-how-prometheu...

Video - https://www.youtube.com/watch?v=b-qLOY5ChnQ

Re: Ask HN: Best monitoring system?

#47

Post by postwait »

falcolas wrote: post #5

Just my opinion, but I won't use Prometheus, because of the active polling model. It won't scale without a number of workarounds. My preferred method is Icinga2 (a Nagios clone with better configuration and clustering built-in) with reports coming in via passive NSCA. Toss in Graphite (or I'm warming up to Grafana on Influx) with some ability to write alerts against those reported metrics, and you're as close to idea…

A more thorough investigation of pull vs. push.

https://www.circonus.com/pully-mcpushface/

Re: Ask HN: Best monitoring system?

#49

Post by dozzie »

falcolas wrote: post #36
sagichmal wrote: post #19

Prometheus is absolutely the way you should be going. All of the other systems I'm seeing mentioned here — Nagios, Icinga, check_mk, Zabbix, Sensu — are host-centric and are very awkward when you try to bend them to fit modern (containerized, etc.) workloads.

There's always a server. Regardless of how far away you've abstracted it away, there's always a server which should probably be monitored (even if to know when it's about to fail and should have work shunted off it prior to its failure). Icinga and others make it easy to programmatically add and remove servers as they enter and leave your environment. Even if you don't have access to the server so you can monitor it,…

> There's always a server.

Actually, no.

When you monitor state of a cluster (e.g. node count), you don't have a server, you have plenty of servers and a cluster (completely different thing).

When you monitor temperature in your server room, you don't have a server, you have a server room.

When you monitor exchange rate, you don't have a server.

When you monitor a website, you still don't have a server.

And now add all the AWS Lambdas and other serverless rage.

Notion that everything works on a (single!) server was never valid, and today it's even more visible than it was twenty years ago, when Nagios was state of the art.

Re: Ask HN: Best monitoring system?

#50

Post by dsr_ »

mspaulding06 wrote: post #25
dsr_ wrote: post #21

It depends on your architecture and scale. There is no "best", just "best we've found for this" and "best given other constraints". This is yet another point where DevOps is not "devs doing ops" but "operations building and deploying with all the tools of modern software development". You need a subject matter expert. What are you monitoring? Do you care about availability or performance or both? Scale? Do you have s…

Thanks for the comprehensive reply. Briefly I would say that we care about availability more than performance, though both are important. We're running somewhere on the order of 2000 VMs w/ ESX with some bare metal systems running database clusters. We have a separate team that manages the hardware infrastructure and they have their own monitoring and alerting system. I'm mostly concerned about preventing downtime fo…

I still don't know anything much about your systems, but I do know this: find out what your hardware team uses and see if it is right for you, too. (Or find out that they are unhappy with it, and perhaps go in together on a new system.)

Benefits: shared expertise. Common language. Propagation of alerts up from hardware and down from services. Better root cause analysis. If you have a good culture, faster resolution time and better understanding.

Post reply on HN