Live data from Hacker News

Show HN: I built an open-source tool to make on-call suck less

github.com

1–10 of 174 posts

Show HN: I built an open-source tool to make on-call suck less

#1
Hey HN,

I am building an open source platform to make on-call better and less stressful for engineers. We are building a tool that can silence alerts and help with debugging and root cause analysis. We also want to automate tedious parts of being on-call (running runbooks manually, answering questions on Slack, dealing with Pagerduty). Here is a quick video of how it works: https://youtu.be/m_K9Dq1kZDw

I hated being on-call for a couple of reasons:

* Alert volume: The number of alerts kept increasing over time. It was hard to maintain existing alerts. This would lead to a lot of noisy and unactionable alerts. I have lost count of the number of times I got woken up by alert that auto-resolved 5 minutes later.

* Debugging: Debugging an alert or a customer support ticket would need me to gain context on a service that I might not have worked on before. These companies used many observability tools that would make debugging challenging. There are always a time pressure to resolve issues quickly.

There were some more tangential issues that used to take up a lot of on-call time

* Support: Answering questions from other teams. A lot of times these questions were repetitive and have been answered before.

* Dealing with PagerDuty: These tools are hard to use. e.g. It was hard to schedule an override in PD or do holiday schedules.

I am building an on-call tool that is Slack-native since that has become the de-facto tool for on-call engineers.

We heard from a lot of engineers that maintaining good alert hygiene is a challenge.

To start off, Opslane integrates with Datadog and can classify alerts as actionable or noisy.

We analyze your alert history across various signals:

1. Alert frequency

2. How quickly the alerts have resolved in the past

3. Alert priority

4. Alert response history

Our classification is conservative and it can be tuned as teams get more confidence in the predictions. We want to make sure that you aren't accidentally missing a critical alert.

Additionally, we generate a weekly report based on all your alerts to give you a picture of your overall alert hygiene.

What’s next?

1. Building more integrations (Prometheus, Splunk, Sentry, PagerDuty) to continue making on-call quality of life better

2. Help make debugging and root cause analysis easier.

3. Runbook automation

We’re still pretty early in development and we want to make on-call quality of life better. Any feedback would be much appreciated!

Show HN: I built an open-source tool to make on-call suck less
github.com

Re: Show HN: I built an open-source tool to make on-call suck less

#3
every time I see notifications in Slack / Telegram it makes me depressed. Text messengers were not designed for this. If you get the "something is wrong" alert it becomes part of history, it won't re-alert you if it's still present. And if you have more than one type of alert it will be lost in history

I guess alerts to messengers are OK as long it's only a couple manually created ones, and there should be a graphical dashboard to learn the rest of problems

Re: Show HN: I built an open-source tool to make on-call suck less

#4
post #3

every time I see notifications in Slack / Telegram it makes me depressed. Text messengers were not designed for this. If you get the "something is wrong" alert it becomes part of history, it won't re-alert you if it's still present. And if you have more than one type of alert it will be lost in history I guess alerts to messengers are OK as long it's only a couple manually created ones, and there should be a graphica…

Yeah, I agree that slack is not the best medium for alerts. I think we it has somewhat become the default in teams is that it makes it easy to collaborate while debugging. I don't know a good way to substitute that and share information.

What strategies have you seen work well?

Re: Show HN: I built an open-source tool to make on-call suck less

#5
Big fan of this direction. The architecture resonates! The base lining is interesting, I'm curious how you think about that, esp for bootstrapping initially + ongoing.

We are working on a variant being used more by investigative teams than IT ops - so think IR, fraud, misinfo, etc - which has similarities but also domain differences. If of interest to someone with an operational infosec background (hunt, IR, secops) , and esp US-based, the Louie.AI team is hiring an SE + principal here.

Re: Show HN: I built an open-source tool to make on-call suck less

#6
Really cool!

Anyone know of a similar alert UI for data/business alarms (eg installs dropping WoW, crashes spiking DoD, etc)?

Something that feeds of Snowflake/BigQuery, but with a similar nice UI so that you can quickly see false positives and silence them.

The tools I’ve used so far (mostly in-house built) have all ended in a spammy slack channel that no one ever checks anymore.

Re: Show HN: I built an open-source tool to make on-call suck less

#7
post #3

every time I see notifications in Slack / Telegram it makes me depressed. Text messengers were not designed for this. If you get the "something is wrong" alert it becomes part of history, it won't re-alert you if it's still present. And if you have more than one type of alert it will be lost in history I guess alerts to messengers are OK as long it's only a couple manually created ones, and there should be a graphica…

Why? We send alerts to Slack and Pagerduty. Slack is to help everyone who might be working, PagerDuty alerts the persons who are actually in charge of working on it.

Re: Show HN: I built an open-source tool to make on-call suck less

#8
> Slack-native since that has become the de-facto tool for on-call engineers.

In your particular organization. Slack is one of many instant messaging platforms. Tightly coupling your tool to Slack instead of making it platform agnostic immediately restricts where it can be used.

Other comment threads are already discussing the broader issues with using IM for this job, so I won't go into it here.

Regardless, well done for making something.

Re: Show HN: I built an open-source tool to make on-call suck less

#9

> Slack-native since that has become the de-facto tool for on-call engineers. In your particular organization. Slack is one of many instant messaging platforms. Tightly coupling your tool to Slack instead of making it platform agnostic immediately restricts where it can be used. Other comment threads are already discussing the broader issues with using IM for this job, so I won't go into it here. Regardless, well don…

Thanks for the feedback. We want to get something out quickly and we had experience working with Slack so it made sense for us to start there.

However, the design is pretty flexible and we don't want to tie ourselves to a single platform either.

Re: Show HN: I built an open-source tool to make on-call suck less

#10
post #3

every time I see notifications in Slack / Telegram it makes me depressed. Text messengers were not designed for this. If you get the "something is wrong" alert it becomes part of history, it won't re-alert you if it's still present. And if you have more than one type of alert it will be lost in history I guess alerts to messengers are OK as long it's only a couple manually created ones, and there should be a graphica…

THIS. Whispering into a slack channel off hours isn’t a way to get on-call support help nor is dropping alerts in one. If it’s a critical issue I’m going to need a page of some kind. Either from something like PagerDuty or directly wired up SMS messaging.
Post reply on HN