Live data from Hacker News

Real-time notifications from systemd to Slack

scaledrone.com

31–40 of 55 posts

Re: Real-time notifications from systemd to Slack

#31
Why is everyone okay with relying on Slack for sensitive communication or even business critical stuff like monitoring?

It lacks everything a good alerting system has (acknowledgements, fine-grained notifications...).

Even those addresses in the stacktraces could be sensitive in other situations since they contain ASLR offsets.

Re: Real-time notifications from systemd to Slack

#32
post #25

What if slack is down/unreachable? Won't it keep the service from starting?

If it makes slacktee fail with an error code, yes, it would prevent startup. The fix is a - sign:

    ExecStartPre=-/usr/bin/foo
Would still mark the service as "failed" but wouldn't prevent it from starting.

Re: Real-time notifications from systemd to Slack

#33
post #31

Why is everyone okay with relying on Slack for sensitive communication or even business critical stuff like monitoring? It lacks everything a good alerting system has (acknowledgements, fine-grained notifications...). Even those addresses in the stacktraces could be sensitive in other situations since they contain ASLR offsets.

What would you suggest? We're in need of an alerting platform and making use of Slack, not because it's flawless or anything, but because we're already paying for it and it's convenient for the mass junk. Ie, not strictly mission critical alerts.

What medium might be better than Slack?

(to be clear, i'm asking, not saying Slack is good for this)

Re: Real-time notifications from systemd to Slack

#34
post #26
post #20

Earlier quoted context omitted.

Interesting. Thank you! Would you mind sharing some of the faults? By instances, are you referring to template unit files like this[0]? Do you use it with the `--user` option or do you just access it with `sudo systemctl`? 0: https://fedoramagazine.org/systemd-template-unit-files/

Yes, I do use --user where possible. Because resource limits are per-user on UNIX, I tend to run every program and every instance of a service as a separate user. cgroups/containers make this less necessary, but old habits die hard. The biggest systemd fault is one of tooling, and that just comes from a project ambitious enough to try and own it's own ecosystem. When runit/daemontools, you "debug" a service by typing…

Ah, hmm. Thank you very much for sharing. Looks like it's got some way before I can use it instead of `monit`. This helps a lot.

Re: Real-time notifications from systemd to Slack

#35
post #31

Why is everyone okay with relying on Slack for sensitive communication or even business critical stuff like monitoring? It lacks everything a good alerting system has (acknowledgements, fine-grained notifications...). Even those addresses in the stacktraces could be sensitive in other situations since they contain ASLR offsets.

Actually, the PagerDuty integration for Slack allows in-Slack acknowledgement (and/or resolution) of alerts.

(we used this in addition to, not in lieu of, the pagerduty dashboard itself, as well as mobile notification via pagerduty, and Nagio's web UI.)

> fine-grained notifications

what do you mean by this, out of curiosity?

Re: Real-time notifications from systemd to Slack

#36
post #13

Can anyone who's used systemd share what it's like to have it for process management? We use monit, and it's all right, but I occasionally wonder. Does anyone run it also as a separate user to manage certain applications? So that you could have certain people log on and operate some things and others operate other things?

systemd is far superior to anything else out there. It matured at roughly the same time as docker. However it doesn't support being the CMD in a Dockerfile. Which is why it's not very common in software deployment scenarios in the post-container world. For older deployments, it may not be worth switching to systemd because the base OS may not be compatible. So it's kind of a catch-22. If you are on baremetal, systemd…

Systemd works inside LXC and systemd-nspawn.

Re: Real-time notifications from systemd to Slack

#37
post #22

Earlier quoted context omitted.

While nohup is used to daemonize a single process, systemd does that and a ton of other stuff. It's a replacement init system that includes everything from daemonization, log handling, auto restarts, and by this time next year will probably make coffee for you as well.

This is what I've never understood about systemd. Somebody will ask "doesn't inetd already do that?" or "doesn't nohup already do that?" or "doesn't runit already do that?" and the answer is always "yes, but systemd does a bunch of other things something else already did too". shrug Its setup is way, way too brittle for me, but I guess the QR codes are kind of neat.

My requirements (for server side daemons) are approximately: start a daemon as a child, and monitor it; if it dies, restart it. Ideally, give me a command-line interface to the managing daemon to gracefully start/stop the child. nohup doesn't do this, making it simply not the right tool for the job. I've never used runit, but it looks closer.

Re: Real-time notifications from systemd to Slack

#38

Earlier quoted context omitted.

This is what I've never understood about systemd. Somebody will ask "doesn't inetd already do that?" or "doesn't nohup already do that?" or "doesn't runit already do that?" and the answer is always "yes, but systemd does a bunch of other things something else already did too". shrug Its setup is way, way too brittle for me, but I guess the QR codes are kind of neat.

My requirements (for server side daemons) are approximately: start a daemon as a child, and monitor it; if it dies, restart it. Ideally, give me a command-line interface to the managing daemon to gracefully start/stop the child. nohup doesn't do this, making it simply not the right tool for the job. I've never used runit, but it looks closer.

Take a look at PIES[1]. I've switched most of my production systems to it. I especially like that it's inittab-compatible.

[1]: http://www.gnu.org.ua/software/pies/

Re: Real-time notifications from systemd to Slack

#39
post #19
post #13

Can anyone who's used systemd share what it's like to have it for process management? We use monit, and it's all right, but I occasionally wonder. Does anyone run it also as a separate user to manage certain applications? So that you could have certain people log on and operate some things and others operate other things?

Yes. I used to use runit/daemontools/inittab. My favourite thing about systemd is that it is increasingly available, and while it has it's faults, it has instances (macros you can use to kickoff a fleet of services easily) and pretty good isolation features. It also has a "systemctl-over-ssh" feature which is quite nice, and which allows you to use an .ssh/authorized_keys file instead of sudo to allow access to certa…

allows you to use an .ssh/authorized_keys file instead of sudo to allow access to certain administrative tasks.

Another way to do this is pam_ssh_agent_auth. Been using it to authenticate to sudo for years on systems that only maintain keys and no passwords.

Re: Real-time notifications from systemd to Slack

#40
post #31

Why is everyone okay with relying on Slack for sensitive communication or even business critical stuff like monitoring? It lacks everything a good alerting system has (acknowledgements, fine-grained notifications...). Even those addresses in the stacktraces could be sensitive in other situations since they contain ASLR offsets.

What would you suggest? We're in need of an alerting platform and making use of Slack, not because it's flawless or anything, but because we're already paying for it and it's convenient for the mass junk. Ie, not strictly mission critical alerts. What medium might be better than Slack? (to be clear, i'm asking, not saying Slack is good for this)

Zabbix. It's open-source and pretty flexible. I maintain my company's Zabbix installation, and I've come to appreciate it (and 3.x finally has a non-ugly UI!).
Post reply on HN