Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built an open-source tool to make on-call suck less

github.com

Post reply on HN
Print view View original

81–90 of 174 posts

Re: Show HN: I built an open-source tool to make on-call suck less

#81

Post by callalex »

unethical_ban wrote: post #73
dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

If the tool works properly, then the value proposition is: Sure, you could spend months trying to fix or tune out every useless alarm type, or try to hack your alert manager/email inbox with filters for things you know will get fixed in a few months - OR, you can use this tool that can quickly classify things as important or not.

If something that triggers a pager takes months to resolve you already work at an organization that is so ossified it will be unwilling to adopt a random bandaid startup product like this.

Re: Show HN: I built an open-source tool to make on-call suck less

#82

Post by callalex »

__turbobrew__ wrote: post #19
solatic wrote: post #11

In my current workplace (BigCo), we know exactly what's wrong with our alert system. We get alerts that we can't shut off, because they (legitimately) represent customer downtime, and whose root cause we either can't identify (lack of observability infrastructure) or can't fix (the fix is non-trivial and management won't prioritize). Running on-call well is a culture problem. You need management to prioritize observa…

I work on a team which runs hyper critical infra on all production machines at BigCo and have the same experience as you. The problem are not the alerts — the alerts actually are catching real problems — the problem is the following: 1. The team is understaffed so sometimes spending a few days root causing an alert is not prioritized 2. When alerts are root caused sometimes the work to fix the root cause is not prior…

When describing infrastructure, words matter. When you describe something as “hyper critical infrastructure” it implies that tens to thousands of human beings will die within seconds of failure of said “hyper critical” infrastructure. The way the rest of your comment is worded implies that’s not what you’re actually describing and makes the words “hyper critical infrastructure” irresponsible for you to use.

I don’t mean to imply there is some kind of failure magnitude competition, I just want to reinforce that software “engineering” already has a huge problem with abject neglect of the learnings that other sign-and-stamp engineering fields have already learned from and fixed. Us code slingers are not in uncharted territory, we just need to learn from our predecessors and peers that build literal bridges and towers and force management to treat our field in the same way.

Re: Show HN: I built an open-source tool to make on-call suck less

#83

Post by unethical_ban »

callalex wrote: post #81
unethical_ban wrote: post #73

Earlier quoted context omitted.

If the tool works properly, then the value proposition is: Sure, you could spend months trying to fix or tune out every useless alarm type, or try to hack your alert manager/email inbox with filters for things you know will get fixed in a few months - OR, you can use this tool that can quickly classify things as important or not.

If something that triggers a pager takes months to resolve you already work at an organization that is so ossified it will be unwilling to adopt a random bandaid startup product like this.

I'm sure that's true in many places. The number of large companies deal with this kind of thing (understaffed teams operating hundreds or thousands of devices) is quite high. Some can pull off shadow IT or exceptions for free software.

Re: Show HN: I built an open-source tool to make on-call suck less

#85

Post by throwaway984393 »

Don't send an alert at all unless it is actionable. Yes, I get it, you want alerts for everything. Do you have a runbook that can explain to a complete novice what is going on and how to fix the problem? No? Then don't alert on it.

The only way to make on-call less stressful is to do the boring work of preparing for incidents, and the boring work of cleaning up after incidents. No magic software will do it for you.

Re: Show HN: I built an open-source tool to make on-call suck less

#86

Post by dambi0 »

callalex wrote: post #82
__turbobrew__ wrote: post #19

Earlier quoted context omitted.

I work on a team which runs hyper critical infra on all production machines at BigCo and have the same experience as you. The problem are not the alerts — the alerts actually are catching real problems — the problem is the following: 1. The team is understaffed so sometimes spending a few days root causing an alert is not prioritized 2. When alerts are root caused sometimes the work to fix the root cause is not prior…

When describing infrastructure, words matter. When you describe something as “hyper critical infrastructure” it implies that tens to thousands of human beings will die within seconds of failure of said “hyper critical” infrastructure. The way the rest of your comment is worded implies that’s not what you’re actually describing and makes the words “hyper critical infrastructure” irresponsible for you to use. I don’t m…

Words matter but so does context. You weren’t confused by the words here why assume others would be?

Re: Show HN: I built an open-source tool to make on-call suck less

#87

Post by Too »

aray07 wrote: post #27
whazor wrote: post #25

Earlier quoted context omitted.

Or maybe page your managers, such that they can escalate the situation. They will be more aligned on solving the cultural problems if they get waked up too.

yeah the best managers i worked with used to be on the same on-call rotation such that they would also get paged every time. That helped build empathy and visibility into the situation.

Wouldn't the manager of one team be part of every shift in such a setup?

Re: Show HN: I built an open-source tool to make on-call suck less

#89

Post by physicsguy »

efxhoy wrote: post #37
solatic wrote: post #11

In my current workplace (BigCo), we know exactly what's wrong with our alert system. We get alerts that we can't shut off, because they (legitimately) represent customer downtime, and whose root cause we either can't identify (lack of observability infrastructure) or can't fix (the fix is non-trivial and management won't prioritize). Running on-call well is a culture problem. You need management to prioritize observa…

> Running on-call well is a culture problem. You need management to prioritize observability (you can't fix what you can't show as being broken), then you need management to build a no-broken-windows culture (feature development stops if anything is broken). I was lucky enough to join a company where management does this. The managers were made to do this by experienced engineers who explained to them in no uncertain…

I tried that approach with a colleague and it just got more and more heated and frustrating. At the same time we were getting heat for reliability. I ended up quitting. Since then I heard from a colleague that they made some staff redundant, on a team that was already underwater.

I doubt very much that my experience was unique. In my new position we have the same problems with reliability but I don’t get involved in the political side of trying to argue about it, just turn up and do my 9-5. I’m a lot less stressed now!

Re: Show HN: I built an open-source tool to make on-call suck less

#90

Post by Too »

dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

100x this. Garbage in = Garbage out.

In similar mindset, I've seen attempts to "fix" flaky test suites by retrying failing tests 5 times until they pass. What happens: You just set a new baseline of shit allowed. This allows even more noise to enter the system and you have to rerun them even more or need increasingly more advanced tools to filter out the noise.

Once the new baseline is anchored, you become dependent on the filter. Now every tool that interacts with the metrics need to be aware of the additional filter, there may be more than just the slack messages. Should your dashboard show the raw or the filtered metrics?

Devils advocate: consider an alert with 99/100 false positives. The LLM may be good at classifying it as noisy but will it do a better job than a human to react to the 1 true positive? Maybe, but at the same time it allows more such noise to accumulate in the system, in effect a net negative. It's better to remove such an alert instead. Even if the numbers were turned around in favor, that's a lot of complexity added.

The additional context this product provides may of course still be useful and i applaud the effort. This product space does have a lot of potential for growth and is a real pain for operators. Be careful with using it as a substitute for proper alert hygiene and culture.

Post reply on HN