Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built an open-source tool to make on-call suck less

github.com

Post reply on HN
Print view View original

11–20 of 174 posts

Re: Show HN: I built an open-source tool to make on-call suck less

#11

Post by solatic »

In my current workplace (BigCo), we know exactly what's wrong with our alert system. We get alerts that we can't shut off, because they (legitimately) represent customer downtime, and whose root cause we either can't identify (lack of observability infrastructure) or can't fix (the fix is non-trivial and management won't prioritize).

Running on-call well is a culture problem. You need management to prioritize observability (you can't fix what you can't show as being broken), then you need management to build a no-broken-windows culture (feature development stops if anything is broken).

Technical tools cannot fix culture problems!

edit: management not talking to engineers, or being aware of problems and deciding not to prioritize fixing them, are both culture problems. The way you fix culture problems, as someone who is not in management, is to either turn your brain off and accept that life is imperfect (i.e. fix yourself instead of the root cause), or to find a different job (i.e. if the culture problem is so bad that it's leading to burnout). In any event, cultural problems cannot be solved with technical tools.

Re: Show HN: I built an open-source tool to make on-call suck less

#12

Post by aray07 »

solatic wrote: post #11

In my current workplace (BigCo), we know exactly what's wrong with our alert system. We get alerts that we can't shut off, because they (legitimately) represent customer downtime, and whose root cause we either can't identify (lack of observability infrastructure) or can't fix (the fix is non-trivial and management won't prioritize). Running on-call well is a culture problem. You need management to prioritize observa…

I completely agree that technical tools cannot fix culture problems.

However, one of the things that I noticed in my previous companies was that my management chain wasn't even aware that the problem was this bad.

We also wanted to add better reporting (like the alert analytics) so that people have more visibility into the state of alerts + on-call load on engineers.

What strategies have worked well for you when it comes to management prioritizing these problems?

Re: Show HN: I built an open-source tool to make on-call suck less

#13

Post by maximinus_thrax »

Nice work, I always appreciate the contribution to the OSS ecosystem.

That said, I like that you're 'saying out loud' with this. Slack and other similar comm tooling has always been advertised as a productivity booster due to their 'async' nature. Nobody actually believes this anymore and coupling it with the oncall notifications really closes the lid on that thing.

Re: Show HN: I built an open-source tool to make on-call suck less

#14

Post by hoistbypetard »

solatic wrote: post #11

In my current workplace (BigCo), we know exactly what's wrong with our alert system. We get alerts that we can't shut off, because they (legitimately) represent customer downtime, and whose root cause we either can't identify (lack of observability infrastructure) or can't fix (the fix is non-trivial and management won't prioritize). Running on-call well is a culture problem. You need management to prioritize observa…

That’s true. But technical tools can help you highlight culture problems so that they’re easier to to discuss and fix. It’s been a minute since I’ve had to process exactly the kind of on-call/alert problem we’re discussing here, but this does feel like the kind of tool that would help sell the kinds of management/culture changes necessary to really improve things, if not fix all of them.

Re: Show HN: I built an open-source tool to make on-call suck less

#15

Post by FooBarWidget »

RadiozRadioz wrote: post #8

> Slack-native since that has become the de-facto tool for on-call engineers. In your particular organization. Slack is one of many instant messaging platforms. Tightly coupling your tool to Slack instead of making it platform agnostic immediately restricts where it can be used. Other comment threads are already discussing the broader issues with using IM for this job, so I won't go into it here. Regardless, well don…

Try Netherlands. We're Microsoft land over here. Pretty much everyone is on Azure and Teams. It's mostly startups and hip small companies that use Slack.

Re: Show HN: I built an open-source tool to make on-call suck less

#16

Post by aray07 »

maximinus_thrax wrote: post #13

Nice work, I always appreciate the contribution to the OSS ecosystem. That said, I like that you're 'saying out loud' with this. Slack and other similar comm tooling has always been advertised as a productivity booster due to their 'async' nature. Nobody actually believes this anymore and coupling it with the oncall notifications really closes the lid on that thing.

Yeah, unfortunately, I don't think these messaging tools are async. During oncall, I used to pretty much live on Slack. Incidents were on slack, customer tickets on slack, debugging on slack...

Re: Show HN: I built an open-source tool to make on-call suck less

#17

Post by cyanydeez »

solatic wrote: post #11

In my current workplace (BigCo), we know exactly what's wrong with our alert system. We get alerts that we can't shut off, because they (legitimately) represent customer downtime, and whose root cause we either can't identify (lack of observability infrastructure) or can't fix (the fix is non-trivial and management won't prioritize). Running on-call well is a culture problem. You need management to prioritize observa…

Obviously, the best way to get management's attention is to start a stop and frisk customer engagement plan.

Re: Show HN: I built an open-source tool to make on-call suck less

#18

Post by lars_francke »

Shameless question tangential related to the topic.

We are based in Europe and have the problem that some of us sometimes just forget we're on call or are afraid that we'll miss OpsGenie notifications.

We're desparately looking for a hardware solution. I'd like something similar to the pagers of the past but at least here in Germany they don't really seem to exist anymore. Ideally I'd have a Bluetooth dongle that alerts me on configurable notifications on my phone. Carrying this dongle for the week would be a physical reminder I'm on call.

Does anyone know anything?

Re: Show HN: I built an open-source tool to make on-call suck less

#19

Post by __turbobrew__ »

solatic wrote: post #11

In my current workplace (BigCo), we know exactly what's wrong with our alert system. We get alerts that we can't shut off, because they (legitimately) represent customer downtime, and whose root cause we either can't identify (lack of observability infrastructure) or can't fix (the fix is non-trivial and management won't prioritize). Running on-call well is a culture problem. You need management to prioritize observa…

I work on a team which runs hyper critical infra on all production machines at BigCo and have the same experience as you.

The problem are not the alerts — the alerts actually are catching real problems — the problem is the following:

1. The team is understaffed so sometimes spending a few days root causing an alert is not prioritized 2. When alerts are root caused sometimes the work to fix the root cause is not prioritized 3. A culture on the team which allows alerts to go untriaged due to desensitization.

Our headcount got reduced by ~40% and — surprise surprise — reliability and on-call got much worse. Senior leadership has made the decision that the cost cuts are worth the decreased reliability so nothing is going to change.

The job market is rough so people put up with this for now.

Re: Show HN: I built an open-source tool to make on-call suck less

#20

Post by dclowd9901 »

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts.

grimace face

I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and we can certainly use more competition in this space, but this aspect of it is basically enabling bad cultural practices and I wouldn’t highlight it as a main selling point.

Post reply on HN