Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built an open-source tool to make on-call suck less

github.com

Post reply on HN
Print view View original

91–100 of 174 posts

Re: Show HN: I built an open-source tool to make on-call suck less

#91

Post by karamanolev »

steve1977 wrote: post #75
DanHulton wrote: post #67

Earlier quoted context omitted.

I agree with your intent and desire, but the fact is that this problem _keeps happening,_ and we can't fix it by advocating "well, just do alerts better." There's a lot of cultural inertia at a lot of places that leads to creating too many, too low-signal alerts, and fixing that across an entire company -- or, hell, industry -- is a magnificently tall order. However, installing a tool to specifically reign in those g…

If you work at a shitty place, focus your energy on leaving the shitty place.

The shittiness of a place is defined by a lot more, and more important, attributes than alert hygiene. Culture, pay, location, industry, leadership. If one leaves companies for (relatively) minor things like that, there are basically no companies left to work for.

Re: Show HN: I built an open-source tool to make on-call suck less

#92

Post by mads_quist »

Founder of All Quiet here: https://allquiet.app.

We're building a tool in the same space but opted out of using LLMs. We've received a lot of positive feedback from our users who explicitly didn't want critical alerts to be dependent on a possibly opaque LLM. While I understand that some teams might choose to go this route, I agree with some commentators here that AI can help with symptoms but doesn't address the root cause, which is often poor observability and processes.

Re: Show HN: I built an open-source tool to make on-call suck less

#93

Post by Jolter »

sanj001 wrote: post #31

Using LLMs to classify noisy alerts is a really clever approach to tackling alert fatigue! Are you fine tuning your own model to differentiate between actionable and noisy alerts? I'm also working on an open source incident management platform called Incidental ( https://github.com/incidentalhq/incidental ), slightly orthogonal to what you're doing, and it's great to see others addressing these on-call challenges. Ou…

I wouldn’t say it’s particularly clever. It’s a fairly obvious idea to anyone who has worked with alerting through IMs. What it is, is /difficult/, because you really really need to avoid false positives. Probably lots of hard work involved. So kudos for making this work (if it works)!

Re: Show HN: I built an open-source tool to make on-call suck less

#94

Post by Jolter »

Telecoms solved this problem fifteen years ago when they started automating Fault Management (google it).

Granted, neural networks were not generally applicable to this problem at the time, but this whole idea seems like the same problem being solved again.

Telecoms and IT used to supervise their networks using Alarms, in either a Network Management System (NMS) or something more ad-hoc like Nagios. There, you got structured alarms over a network, like SNMP traps, that got stored as records in a database. It’s fairly easy to program filters using simple counting or more complex heuristics against a database.

Now, for some reason, alerting has shifted to Slack. Naturally since the data is now unstructured text, the solution involves an LLM! You build complexity into the filtering solution because you have an alarm infrastructure that’s too simple.

Re: Show HN: I built an open-source tool to make on-call suck less

#95

Post by gklitz »

dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

> enabling bad cultural practices

I strongly disagree. There is nothing culturally bad in a system issuing an error if there is an error. Sometimes systems issue errors that are considered noise by supporters because they are not actionable, but forcing a system to not issue an error just because your support team cannot directly take action on it is an extremely odd leakage of team responsibilities and bound to have unintended consequences. Imagine a developer telling management that they didn’t implement error checking on some edge case because the support team told them they didn’t have documentation about how to take action for instance. The appropriate response there would be “why on earth are you asking support permission to add error messages for a known error?”. On the other hand, if a support team is drowning in noisy error messages they need tooling to make it easy to distinguish between those and other messages that need to be reviewed of have action taken.

Re: Show HN: I built an open-source tool to make on-call suck less

#96

Post by remus »

dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

To give a more charitable take, in practice it's easy for noisy alerts to creep in and a tool like this could be a good nudge to dig a bit deeper on that alert and why it's gone from useful to noise.

Re: Show HN: I built an open-source tool to make on-call suck less

#97

Post by remus »

gklitz wrote: post #95
dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

> enabling bad cultural practices I strongly disagree. There is nothing culturally bad in a system issuing an error if there is an error. Sometimes systems issue errors that are considered noise by supporters because they are not actionable, but forcing a system to not issue an error just because your support team cannot directly take action on it is an extremely odd leakage of team responsibilities and bound to have…

> There is nothing culturally bad in a system issuing an error if there is an error.

That's true, but if the error says "PANIC! EVERYTHING IS DOWN" when it's not true, then it's asking for an action that's outsized to the problem. Error messages are fine, but they just need to be classified and responded to correctly, and noisy alerts are typically the ones that are misclassified and demanding attention they (probably) don't deserve.

Re: Show HN: I built an open-source tool to make on-call suck less

#98

Post by lazyasciiart »

lupire wrote: post #57
__turbobrew__ wrote: post #19

Earlier quoted context omitted.

I work on a team which runs hyper critical infra on all production machines at BigCo and have the same experience as you. The problem are not the alerts — the alerts actually are catching real problems — the problem is the following: 1. The team is understaffed so sometimes spending a few days root causing an alert is not prioritized 2. When alerts are root caused sometimes the work to fix the root cause is not prior…

It sounds like you forget to make an SLO? If an alert is not actionable because it's impossible to resolve, even though it has customer impact, then it should be an SLO, not an alert.

SLO as in “service level objective”? How does defining an SLO stop the existence of alerts?

Re: Show HN: I built an open-source tool to make on-call suck less

#99

Post by lars_francke »

Geezus_42 wrote: post #69
jobtemp wrote: post #39

Earlier quoted context omitted.

There are phone apps that can pierce through all silent or DND settings. Get one of those. If the same app could buzz on less than 50% battery to remind to charge that would help. Also same app could request to confirm on call status so the don't forget. If they don't confirm someone else gets the shift.

OpsGenie has that. We use it at my job. I'm not sure what problem OP is having. The phone call, text message, and app alert from OpsGenie are more than enough. The notification configuration is extremely flexible and each user can customize it as needed. From a user perspective, I don't know what else you could want. I have no affiliation with OpsGenie outside of using at work.

We have OnePlus users with issues: https://support.atlassian.com/opsgenie-android/docs/troubles...

And we have one user with another brand that also locks down the notification/alert settings and kills apps in attempt "to save battery" which can't be controlled.

Re: Show HN: I built an open-source tool to make on-call suck less

#100

Post by lars_francke »

linuxdude314 wrote: post #52
lars_francke wrote: post #18

Shameless question tangential related to the topic. We are based in Europe and have the problem that some of us sometimes just forget we're on call or are afraid that we'll miss OpsGenie notifications. We're desparately looking for a hardware solution. I'd like something similar to the pagers of the past but at least here in Germany they don't really seem to exist anymore. Ideally I'd have a Bluetooth dongle that ale…

This really sounds like a _you_ problem and not something you need hardware to fix. You can already enable silence/focus time bypass modes for apps like PagerDuty and such… If you can’t develop some sense of responsibility to check if you’re on-call, frankly you have no business being in an on-call role. No hardware will make you or your engineers more diligent. The only reason pagers made more sense than phones is/w…

Yup, this is absolutely a "me" problem. Life just gets in the way. I'm on call, the kids ask to go to the pool, I leave my phone in the locker forgetting that I'm on-call. I visit friends, forget to bring my laptop etc.

I realize it's a "me" problem and therefore I'm looking for a solution. Others in the company have the same problem. That said: This is my very own company and I have a great sense of responsibility but I also have a shit-ton of other things in my head and I'm not the only one with this issue here.

The silence etc. bypass doesn't always work (I commented in another thread).

Post reply on HN