Real-time notifications from systemd to Slack • Hacker News Board

Re: Real-time notifications from systemd to Slack

#41

Post by lima » Mon, Mar 13, 2017, 3:33 PM UTC

deathanatos wrote: post #35

lima wrote: post #31

Why is everyone okay with relying on Slack for sensitive communication or even business critical stuff like monitoring? It lacks everything a good alerting system has (acknowledgements, fine-grained notifications...). Even those addresses in the stacktraces could be sensitive in other situations since they contain ASLR offsets.

Actually, the PagerDuty integration for Slack allows in-Slack acknowledgement (and/or resolution) of alerts. (we used this in addition to, not in lieu of, the pagerduty dashboard itself, as well as mobile notification via pagerduty, and Nagio's web UI.) > fine-grained notifications what do you mean by this, out of curiosity?

With PagerDuty on top it's fine from an operational point of view (you'll still have to trust both PagerDuty and Slack to keep your data secure).

> fine-grained notifications

A common approach is having a "alerts" channel that grabs everyones attention, even though most alerts are only relevant for a small subset of people. A company-wide #general channel is bad enough, and an #alerts channel only makes it worse.

You can totally do this with Slack by having something like PagerDuty in-between. We use a custom-written alerting broker that makes it easier to correctly handle some of the more complicated cases but it's pretty much the same idea.

Re: Real-time notifications from systemd to Slack

#42

Post by lima » Mon, Mar 13, 2017, 3:41 PM UTC

amyjess wrote: post #40

notheguyouthink wrote: post #33

Earlier quoted context omitted.

What would you suggest? We're in need of an alerting platform and making use of Slack, not because it's flawless or anything, but because we're already paying for it and it's convenient for the mass junk. Ie, not strictly mission critical alerts. What medium might be better than Slack? (to be clear, i'm asking, not saying Slack is good for this)

Zabbix. It's open-source and pretty flexible. I maintain my company's Zabbix installation, and I've come to appreciate it (and 3.x finally has a non-ugly UI!).

We use Nagios + CheckMK, but I believe neither this nor Zabbix solve the alerting part.

Our solution is a custom notification broker that decides whom to alert and then waits for an acknowledgement. It uses different backends including our company chat.

Not complicated at all, just 100 lines of Python code that contain the business logic.

Anything that relies on a single medium is unsuitable for anything but unimportant alerts. What if Slack goes down for 2 hour? Unlikely, but definitely possible.

This ensures that every alert is explicitly acknowledged by someone, and that unimportant alerts are quickly forgotten without wondering whether someone handled them or not.

We have different applications sending alerts, not just Nagios (because Nagios sucks at processing events as opposed to states), and it would quickly become unmanageable without some sort of middleware.

Re: Real-time notifications from systemd to Slack

#43

Post by stephenr » Mon, Mar 13, 2017, 3:53 PM UTC

sandGorgon wrote: post #29

arjie wrote: post #13

Can anyone who's used systemd share what it's like to have it for process management? We use monit, and it's all right, but I occasionally wonder. Does anyone run it also as a separate user to manage certain applications? So that you could have certain people log on and operate some things and others operate other things?

systemd is far superior to anything else out there. It matured at roughly the same time as docker. However it doesn't support being the CMD in a Dockerfile. Which is why it's not very common in software deployment scenarios in the post-container world. For older deployments, it may not be worth switching to systemd because the base OS may not be compatible. So it's kind of a catch-22. If you are on baremetal, systemd…

> It matured at roughly the same time as docker.

Docker, "matured", in past tense? Proof of time travel right there!

Re: Real-time notifications from systemd to Slack

#44

Post by chimeracoder » Mon, Mar 13, 2017, 4:19 PM UTC

sandGorgon wrote: post #29

arjie wrote: post #13

Can anyone who's used systemd share what it's like to have it for process management? We use monit, and it's all right, but I occasionally wonder. Does anyone run it also as a separate user to manage certain applications? So that you could have certain people log on and operate some things and others operate other things?

systemd is far superior to anything else out there. It matured at roughly the same time as docker. However it doesn't support being the CMD in a Dockerfile. Which is why it's not very common in software deployment scenarios in the post-container world. For older deployments, it may not be worth switching to systemd because the base OS may not be compatible. So it's kind of a catch-22. If you are on baremetal, systemd…

> However it doesn't support being the CMD in a Dockerfile. Which is why it's not very common in software deployment scenarios in the post-container world.

Er, that's only half-true. systemd isn't great for running as PID 1 inside a Dockerfile, but that's because Docker already monitors PID 1[0], and systemd can be used to monitor your container itself.

In other words, think of containers as individual applications that you want to monitor, and systemd can be used either to monitor them or even to run the containers directly. (Yes, systemd can even run Docker containers directly, without Docker![1])

[0] you are using exec mode, right?

[1] https://chimeracoder.github.io/docker-without-docker/#1

Re: Real-time notifications from systemd to Slack

#45

Post by tannhaeuser » Mon, Mar 13, 2017, 5:09 PM UTC

lorenzhs wrote: post #5

Not to be overly nitpicky here but it's spelled "systemd", not "SystemD". From its website: Yes, it is written systemd, not system D or System D, or even SystemD. And it isn't system d either. Why? Because it's a system daemon, and under Unix/Linux those are in lower case, and get suffixed with a lower case d. And since systemd manages the system, it's called systemd. It's that simple. https://www.freedesktop.org/wik…

Insisting on Unix naming conventions is odd when SystemD tramples over more important Unix system concepts left and right.

Re: Real-time notifications from systemd to Slack

#46

Post by daveguy » Mon, Mar 13, 2017, 5:51 PM UTC

lima wrote: post #42

amyjess wrote: post #40

Earlier quoted context omitted.

Zabbix. It's open-source and pretty flexible. I maintain my company's Zabbix installation, and I've come to appreciate it (and 3.x finally has a non-ugly UI!).

We use Nagios + CheckMK, but I believe neither this nor Zabbix solve the alerting part. Our solution is a custom notification broker that decides whom to alert and then waits for an acknowledgement. It uses different backends including our company chat. Not complicated at all, just 100 lines of Python code that contain the business logic. Anything that relies on a single medium is unsuitable for anything but unimport…

Is it something where the business logic part could be easily abstracted / separated? It sounds like an interesting and useful yet simple tool. The open source community can always use more of those.

Edit: or maybe something like a blog post to describe the structural details.

Re: Real-time notifications from systemd to Slack

#47

Post by praveshjain » Mon, Mar 13, 2017, 6:19 PM UTC

jon-wood wrote: post #17

praveshjain wrote: post #14

I don't understand. How is it better than using HTTP POST to post to the slack channel yourself?

It doesn't require changing the underlying systems that are being managed, giving it a lot more flexibility. So long as the service can log to STDOUT it can be integrated with Slack.

Oh I get it. Thanks.

Re: Real-time notifications from systemd to Slack

#48

Post by pavel_lishin » Mon, Mar 13, 2017, 6:24 PM UTC

lima wrote: post #41

deathanatos wrote: post #35

Earlier quoted context omitted.

Actually, the PagerDuty integration for Slack allows in-Slack acknowledgement (and/or resolution) of alerts. (we used this in addition to, not in lieu of, the pagerduty dashboard itself, as well as mobile notification via pagerduty, and Nagio's web UI.) > fine-grained notifications what do you mean by this, out of curiosity?

With PagerDuty on top it's fine from an operational point of view (you'll still have to trust both PagerDuty and Slack to keep your data secure). > fine-grained notifications A common approach is having a "alerts" channel that grabs everyones attention, even though most alerts are only relevant for a small subset of people. A company-wide #general channel is bad enough, and an #alerts channel only makes it worse. You…

If your company is large enough to have alerts that aren't universally applicable, it makes sense to create different channels for different alerts, grouped however you want.

Re: Real-time notifications from systemd to Slack

#49

Post by lima » Mon, Mar 13, 2017, 7:32 PM UTC

pavel_lishin wrote: post #48

lima wrote: post #41

Earlier quoted context omitted.

With PagerDuty on top it's fine from an operational point of view (you'll still have to trust both PagerDuty and Slack to keep your data secure). > fine-grained notifications A common approach is having a "alerts" channel that grabs everyones attention, even though most alerts are only relevant for a small subset of people. A company-wide #general channel is bad enough, and an #alerts channel only makes it worse. You…

If your company is large enough to have alerts that aren't universally applicable, it makes sense to create different channels for different alerts, grouped however you want.

Totally agree, it's exactly what we do. The broker sits in between and decide which alert goes where.

Re: Real-time notifications from systemd to Slack

#50

Post by lima » Mon, Mar 13, 2017, 7:33 PM UTC

daveguy wrote: post #46

lima wrote: post #42

Earlier quoted context omitted.

We use Nagios + CheckMK, but I believe neither this nor Zabbix solve the alerting part. Our solution is a custom notification broker that decides whom to alert and then waits for an acknowledgement. It uses different backends including our company chat. Not complicated at all, just 100 lines of Python code that contain the business logic. Anything that relies on a single medium is unsuitable for anything but unimport…

Is it something where the business logic part could be easily abstracted / separated? It sounds like an interesting and useful yet simple tool. The open source community can always use more of those. Edit: or maybe something like a blog post to describe the structural details.

Would need some clean-up but sure, why not. Ask again in a month or so :)