131–140 of 174 posts
I feel like this would be a great tool for people who have had a much better experience of On Call than I have had. I once worked for a string of businesses that would just send everything to on call unless engineers threatened to quit. Promised automated late night customer sign ups? Haven't actually invested in the website so that it can do that? Just make the on call engineer do it. Too lazy to hire off shore L1 t…
> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…
My company (like so many) is struggling a bit with culture around noisy alarm. Not only is noise tolerated, but when some closes an alarm because it's "known to be noise" and I prod them, it turns out that there is a very real impact on the user, it's just that nobody bothered to look into it. The alarm rings, the on-call hopes that it closes itself soon enough, it does so they consider it "false positive and noisy" even though there was impact on the user during these few minutes.
The only way to fight that is a zero-tolerance culture on alarms, which means no false positive is ever tolerated: fix it.
Telecoms solved this problem fifteen years ago when they started automating Fault Management (google it). Granted, neural networks were not generally applicable to this problem at the time, but this whole idea seems like the same problem being solved again. Telecoms and IT used to supervise their networks using Alarms, in either a Network Management System (NMS) or something more ad-hoc like Nagios. There, you got st…
The goal for oncall should be to NEVER get called. If someone gets called when they are oncall their #1 task the next day is to make sure that call never happens again. That means either fixing a false alarm or tracking down the root cause of the call. Eventually you get to a state where being called is by far the exception instead of the norm.
The goal for oncall should be to NEVER get called. If someone gets called when they are oncall their #1 task the next day is to make sure that call never happens again. That means either fixing a false alarm or tracking down the root cause of the call. Eventually you get to a state where being called is by far the exception instead of the norm.
> Slack-native since that has become the de-facto tool for on-call engineers. In your particular organization. Slack is one of many instant messaging platforms. Tightly coupling your tool to Slack instead of making it platform agnostic immediately restricts where it can be used. Other comment threads are already discussing the broader issues with using IM for this job, so I won't go into it here. Regardless, well don…
As Slack is not end to end encrypted we and I imagine many other companies cannot use it.
Earlier quoted context omitted.
I work on a team which runs hyper critical infra on all production machines at BigCo and have the same experience as you. The problem are not the alerts — the alerts actually are catching real problems — the problem is the following: 1. The team is understaffed so sometimes spending a few days root causing an alert is not prioritized 2. When alerts are root caused sometimes the work to fix the root cause is not prior…
When describing infrastructure, words matter. When you describe something as “hyper critical infrastructure” it implies that tens to thousands of human beings will die within seconds of failure of said “hyper critical” infrastructure. The way the rest of your comment is worded implies that’s not what you’re actually describing and makes the words “hyper critical infrastructure” irresponsible for you to use. I don’t m…
At Netflix we built a feature into our alert systems that added a simple button at the top of every alert that said, "Was this alert useful?". Then we would send the alert owners reports about what percent of people found their alert useful.
It really let us narrow in on which alerts were most useful so that others could subscribe the them, and which were noise, so they could be tuned or shut off.
That one button alone made a huge difference in people's happiness with being on call.
> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…
> which upon investigation might not even be “just” noise My company (like so many) is struggling a bit with culture around noisy alarm. Not only is noise tolerated, but when some closes an alarm because it's "known to be noise" and I prod them, it turns out that there is a very real impact on the user, it's just that nobody bothered to look into it. The alarm rings, the on-call hopes that it closes itself soon enoug…
If we completely shut off the alarm, one time it's actually an issue, I'll get dinged for shutting down monitoring.
So yea, I mark it's priority as low so it doesn't wake me up in PagerDuty and move on.