Show HN: I built an open-source tool to make on-call suck less • Hacker News Board

Re: Show HN: I built an open-source tool to make on-call suck less

#71

Post by Aeolun » Mon, Jul 29, 2024, 2:55 AM UTC

djbusby wrote: post #49

Aeolun wrote: post #42

Earlier quoted context omitted.

> Management needs an education - and that is part of the engineering job Isn’t that bizarre? In all my years as an engineer I can count the number of managers that went to learn about engineering by themselves, on one hand. It’s literally their job, but somehow they feel they can do it without understanding it.

I don't think it is bizarre. I see lots of MBAs running things. They don't have the engineering background, they have the "resources management" background. I think engineer brings the numbers to management to decide course. I prefer the situation where the CTO has no MBA and worked their way up - but that is uncommon IME. So, in many orgs, engineer puts their comms hat on an presents a solid case. The engineer who c…

> I don't think it is bizarre. I see lots of MBAs running things. They don't have the engineering background, they have the "resources management" background.

That part is fine. What I do not understand is why there is so little interest in learning what makes engineering different from running a widgets factory.

“Tell me why it won’t work” is a fine question, but it’d be nice if I didn’t have to force all their education on them.

E.g. how many managers ignore that oft repeated adage that 9 women cannot have a baby in a month, and just spam more people on a project in the hope it’ll go faster.

Re: Show HN: I built an open-source tool to make on-call suck less

#72

Post by fragmede » Mon, Jul 29, 2024, 2:55 AM UTC

Flop7331 wrote: post #54

Is this for missile defense systems or something? What's possibly so important that you need to be woken up for it?

On the Internet, with the war in Ukraine, that's entirely possible!

Re: Show HN: I built an open-source tool to make on-call suck less

#73

Post by unethical_ban » Mon, Jul 29, 2024, 3:09 AM UTC

dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

If the tool works properly, then the value proposition is:

Sure, you could spend months trying to fix or tune out every useless alarm type, or try to hack your alert manager/email inbox with filters for things you know will get fixed in a few months - OR, you can use this tool that can quickly classify things as important or not.

Re: Show HN: I built an open-source tool to make on-call suck less

#74

Post by aray07 » Mon, Jul 29, 2024, 3:29 AM UTC

stitched2gethr wrote: post #68

dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

If I may try to counter this point. You are right, but there's ideal state and there's the real world. When on call most of my time is spent trying to make on call better. Reducing noise, providing more context when the alert and logs are lacking, and of course fixing the real issues that alerts have identified. That said there is a period of time in between receiving non-actionable alerts and classifying them as suc…

Thanks for the feedback. Yeah, ideally, teams would go back and fix their misconfigured alerts. Unfortunately, on-call ends and people forget. The aim was to both provide context when an alert comes up as well as provide a report at the end so everybody has context into the state of alerts.

Re: Show HN: I built an open-source tool to make on-call suck less

#75

Post by steve1977 » Mon, Jul 29, 2024, 4:26 AM UTC

DanHulton wrote: post #67

dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

I agree with your intent and desire, but the fact is that this problem _keeps happening,_ and we can't fix it by advocating "well, just do alerts better." There's a lot of cultural inertia at a lot of places that leads to creating too many, too low-signal alerts, and fixing that across an entire company -- or, hell, industry -- is a magnificently tall order. However, installing a tool to specifically reign in those g…

If you work at a shitty place, focus your energy on leaving the shitty place.

Re: Show HN: I built an open-source tool to make on-call suck less

#76

Post by agent13 » Mon, Jul 29, 2024, 4:28 AM UTC

linuxdude314 wrote: post #52

lars_francke wrote: post #18

Shameless question tangential related to the topic. We are based in Europe and have the problem that some of us sometimes just forget we're on call or are afraid that we'll miss OpsGenie notifications. We're desparately looking for a hardware solution. I'd like something similar to the pagers of the past but at least here in Germany they don't really seem to exist anymore. Ideally I'd have a Bluetooth dongle that ale…

This really sounds like a _you_ problem and not something you need hardware to fix. You can already enable silence/focus time bypass modes for apps like PagerDuty and such… If you can’t develop some sense of responsibility to check if you’re on-call, frankly you have no business being in an on-call role. No hardware will make you or your engineers more diligent. The only reason pagers made more sense than phones is/w…

Wow, stackoverflow vibes here.

Re: Show HN: I built an open-source tool to make on-call suck less

#77

Post by jgraettinger1 » Mon, Jul 29, 2024, 4:41 AM UTC

steve1977 wrote: post #75

DanHulton wrote: post #67

Earlier quoted context omitted.

I agree with your intent and desire, but the fact is that this problem _keeps happening,_ and we can't fix it by advocating "well, just do alerts better." There's a lot of cultural inertia at a lot of places that leads to creating too many, too low-signal alerts, and fixing that across an entire company -- or, hell, industry -- is a magnificently tall order. However, installing a tool to specifically reign in those g…

If you work at a shitty place, focus your energy on leaving the shitty place.

Or, it can be self-evident to all that today's work can deliver more benefit if focused elsewhere, instead of root-causing fickle alert flakes. Customers buy products and services, not alert hygiene.

Re: Show HN: I built an open-source tool to make on-call suck less

#78

Post by darkstar_16 » Mon, Jul 29, 2024, 4:48 AM UTC

FooBarWidget wrote: post #15

RadiozRadioz wrote: post #8

> Slack-native since that has become the de-facto tool for on-call engineers. In your particular organization. Slack is one of many instant messaging platforms. Tightly coupling your tool to Slack instead of making it platform agnostic immediately restricts where it can be used. Other comment threads are already discussing the broader issues with using IM for this job, so I won't go into it here. Regardless, well don…

Try Netherlands. We're Microsoft land over here. Pretty much everyone is on Azure and Teams. It's mostly startups and hip small companies that use Slack.

Denmark is the same. Only the smaller startups use Slack. Everyone one else is on Teams.

Re: Show HN: I built an open-source tool to make on-call suck less

#79

Post by renger6002 » Mon, Jul 29, 2024, 4:49 AM UTC

dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

Agreed. This doesn't make the problem better, it's a bandaid solution that can make the problem worse by allowing you to ignore it for longer.

Iterating on your alarms is super informative about the underlying product. It'll point to how you might improve your KPI measurements, or find bugs you didn't know were there.

Re: Show HN: I built an open-source tool to make on-call suck less

#80

Post by devjab » Mon, Jul 29, 2024, 5:39 AM UTC

dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

As someone who has worked in non-tech startup transitioning into enterprise, or enterprise organisarions for decades… I can tell you that non of these companies or organisations are capable of making meaningful interactions with operations. Disclaimer I haven’t worked operations, but I’ve been in relative close proximity with them work wise for natural reasons or using their infrastructure and sometimes making their tools or helping them with things like automation and Powershell.

Anyway in lot of places management see IT as a necessary evil. Like a service center similar to HR but less popular because management genuinely don’t understand it and most IT departments lack HRs political shrewdness and communication abilities. At the same time it’s not uncommon for users to be unable to tell support if they’re on an Android or iOS device (yes, I’m serious). Sometimes employees won’t even differentiate between their professional and personal IT issues on their work devices. Which means that sometimes they’ll raise hell over things that might not warrant full alert systems for on-site support.

What might be challenging here is that you’ll still need someone to actually use the authors tool correctly. Though that is probably going to be a lot easier than making any sort of change management to was organisations relationship with IT.