111–120 of 174 posts
I don't want to be relying on another flaky LLM for anything mission critical like this. Just fix the original problem, don't layer an LLM into it.
We wanted to provide that awareness because a lot of teams arent fully aware how bad the problem might be (on-calls change weekly, there might be a bunch of other issues)
Is this for missile defense systems or something? What's possibly so important that you need to be woken up for it?
System goes down or degrades in some other way at night and important customers with a different timezone get angry, threatening to leave? (happened with us a few times) But I would'nt use LLM for it due to hallucinations
I don't want to be relying on another flaky LLM for anything mission critical like this. Just fix the original problem, don't layer an LLM into it.
Earlier quoted context omitted.
System goes down or degrades in some other way at night and important customers with a different timezone get angry, threatening to leave? (happened with us a few times) But I would'nt use LLM for it due to hallucinations
What's so important that your customers in other timezones feel like waking you up? If they're ready to walk that fast, you can't trust them not to ditch you for an alternative as soon as they find one.
You don't have to be a missile defense system to require a stable system where devs respond quickly...
It feels to me that using LLM to classify alerts as noisy is just adding risk instead of fixing the root cause of the problem. If an alert is known to be noisy and have appeared on slack before (which is how the LLM would figure out it's a noisy alert), then just remove the alert? Otherwise, how will the LLM know it's noise? Either it will correctly annoy you or hallucinate a reason it figures that alert is just nois…
yeah, thats the goal of adding the context and the report - to hopefully bring awareness to the team that this alert should be removed. My rationale for flagging the alert was to help prioritization for the on-call (lets say there are multiple alerts going off at the same time)
every time I see notifications in Slack / Telegram it makes me depressed. Text messengers were not designed for this. If you get the "something is wrong" alert it becomes part of history, it won't re-alert you if it's still present. And if you have more than one type of alert it will be lost in history I guess alerts to messengers are OK as long it's only a couple manually created ones, and there should be a graphica…
Yeah, I agree that slack is not the best medium for alerts. I think we it has somewhat become the default in teams is that it makes it easy to collaborate while debugging. I don't know a good way to substitute that and share information. What strategies have you seen work well?
I have no idea what's the best setup for small companies
Earlier quoted context omitted.
What's so important that your customers in other timezones feel like waking you up? If they're ready to walk that fast, you can't trust them not to ditch you for an alternative as soon as they find one.
Well, I guess it depends on the business. I forgot to mention that we're B2B. For example, suppose a large food chain or a major bank has an important exam scheduled for their employees on a specific day. If our platform has a blocking bug, no one can proceed (some may be sitting in the class) because the developers are too busy sleeping. Some of our clients are also airplane pilot certification authorities, which ha…
can you build a cheaper datadog instead?
Did you search tools that are cheaper than DD?
Using LLMs to classify noisy alerts is a really clever approach to tackling alert fatigue! Are you fine tuning your own model to differentiate between actionable and noisy alerts? I'm also working on an open source incident management platform called Incidental ( https://github.com/incidentalhq/incidental ), slightly orthogonal to what you're doing, and it's great to see others addressing these on-call challenges. Ou…
Earlier quoted context omitted.
yeah, thats the goal of adding the context and the report - to hopefully bring awareness to the team that this alert should be removed. My rationale for flagging the alert was to help prioritization for the on-call (lets say there are multiple alerts going off at the same time)
That’s a people problem and you cannot fix people problems with tech. If no one cares to do the good job of managing alerts putting AI in front of it will not change that.