Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built an open-source tool to make on-call suck less

github.com

Post reply on HN
Print view View original

121–130 of 174 posts

Re: Show HN: I built an open-source tool to make on-call suck less

#121

Post by everforward »

gklitz wrote: post #95
dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

> enabling bad cultural practices I strongly disagree. There is nothing culturally bad in a system issuing an error if there is an error. Sometimes systems issue errors that are considered noise by supporters because they are not actionable, but forcing a system to not issue an error just because your support team cannot directly take action on it is an extremely odd leakage of team responsibilities and bound to have…

Then it either shouldn't be an alert (and instead part of some kind of summary report or some such) or the devs need to take on call. It is an exercise in frustration for everyone to route the page to ops just to make ops call dev; that means dev still has to have an oncall rotation, they might as well just take the page directly.

The unintended consequence of forcing alerts down ops' throat is them gradually caring less about pages, because there's a very good chance that each one is unactionable. I've worked places that do this, I've seen it happen first-hand more than once.

It starts with frustration and ops being less helpful to devs, and ends in a jaded acceptance where ops people start telling each other "just close it and see if it happens a second or third time, that alert never means anything". At that point, the system may as well not emit the errors anyways because no one is looking at the alerts anyways.

Re: Show HN: I built an open-source tool to make on-call suck less

#122

Post by [deleted] »

CPLX wrote: post #103
renger6002 wrote: post #79

Earlier quoted context omitted.

Agreed. This doesn't make the problem better, it's a bandaid solution that can make the problem worse by allowing you to ignore it for longer. Iterating on your alarms is super informative about the underlying product. It'll point to how you might improve your KPI measurements, or find bugs you didn't know were there.

Bandaids are a valuable and useful product used billions of times around the globe every year.

[deleted]

Re: Show HN: I built an open-source tool to make on-call suck less

#123

Post by ozim »

jrochkind1 wrote: post #120
ozim wrote: post #115

Earlier quoted context omitted.

That’s a people problem and you cannot fix people problems with tech. If no one cares to do the good job of managing alerts putting AI in front of it will not change that.

An AI could help bring to your attention alerts that need managing. I like it for this better than for someone in the moment of receiving an alert deciding whether or not to pay it attention.

If someone ignores alerts they will keep ignoring them but now you automated part of ignoring with "AI" and human at the end still will ignore alerts the same.

Writing it out makes me laugh because that's like something from Douglas Adams stories. Automated Ignoring System along with Infinite Improbability Drive.

Re: Show HN: I built an open-source tool to make on-call suck less

#125

Post by jofer »

aflag wrote: post #108

It feels to me that using LLM to classify alerts as noisy is just adding risk instead of fixing the root cause of the problem. If an alert is known to be noisy and have appeared on slack before (which is how the LLM would figure out it's a noisy alert), then just remove the alert? Otherwise, how will the LLM know it's noise? Either it will correctly annoy you or hallucinate a reason it figures that alert is just nois…

There is a lot to be said for "smoke test" metrics. Things you expect to have frequent false positives, but are sometimes early indicators of larger problems or indicators of where to look deeper if something else goes sideways. They're not things that should wake you up in the middle of the night, but they're a damn valuable tool to quickly figure out what's actually wrong when a "real" alert triggers.

Many of these lend themselves well to dashboards instead of alerts, but not everything is "dashboardable". Sometimes it's good to have a set of low-priority alerts that are treated differently than others.

E.g. "we're not receiving any data/requests". Sometimes that's just a lull in activity. Maybe a holiday. Sometimes it's because everything _else_ is broken and nothing is getting in (e.g. DNS issues).

With that said, I do think that classification should be made manually and not automatically.

Re: Show HN: I built an open-source tool to make on-call suck less

#126

Post by digging »

ozim wrote: post #115
aray07 wrote: post #110

Earlier quoted context omitted.

yeah, thats the goal of adding the context and the report - to hopefully bring awareness to the team that this alert should be removed. My rationale for flagging the alert was to help prioritization for the on-call (lets say there are multiple alerts going off at the same time)

That’s a people problem and you cannot fix people problems with tech. If no one cares to do the good job of managing alerts putting AI in front of it will not change that.

> you cannot fix people problems with tech

For very specific values of "people problem", "fix", and "tech". In reality, a more true (and relevant) assertion is "appropriate tools can make virtually any problem more tractable."

For example, it takes an annoyed engineer to notice that the same flaky alert keeps going off and is noise. Then it takes non-trivial skill on their part to communicate the need to disable that alert. They will meet non-trivial resistance, because disabling alerts is dangerous. However, if the tools they are using say "This is a noisy alert, it hasn't been useful for 6 months," disabling that alert becomes more of a best practice for the organization.

Re: Show HN: I built an open-source tool to make on-call suck less

#128

Post by Terretta »

Note that according to StackOverflows dev survey, more devs use Teams than Slack, over 50% were in Teams. (The stat was called popularity but really should have been prevalence, since a related stat showed devs hated Teams even more than they hated Slack.) Teams has APIs too, and with Microsoft Graph working you can do a lot more than just Teams for them.

More importantly, and not mentioned by StackOverflow, those devs are among the 85% of businesses using M365, meaning they have "Sign in with Microsoft" and are on teams that will pay. The rest have Google and/or Github.

This means despite being a high value hacking target (accounts and passwords of people who operate infrastructure, like the person owned from Snowflake last quarter) you don't have to store passwords therefore can't end up on Have I Been Pwned.

Re: Show HN: I built an open-source tool to make on-call suck less

#129

Post by tasn »

aflag wrote: post #108

It feels to me that using LLM to classify alerts as noisy is just adding risk instead of fixing the root cause of the problem. If an alert is known to be noisy and have appeared on slack before (which is how the LLM would figure out it's a noisy alert), then just remove the alert? Otherwise, how will the LLM know it's noise? Either it will correctly annoy you or hallucinate a reason it figures that alert is just nois…

I like incident.io's take on LLMs with incident management: which is essentially assist, don't decide. [1]

1: https://5x9s.svix.com/p/evolution-of-incident-management

Post reply on HN