Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built an open-source tool to make on-call suck less

github.com

Post reply on HN
Print view View original

21–30 of 174 posts

Re: Show HN: I built an open-source tool to make on-call suck less

#21

Post by maximinus_thrax »

aray07 wrote: post #16
maximinus_thrax wrote: post #13

Nice work, I always appreciate the contribution to the OSS ecosystem. That said, I like that you're 'saying out loud' with this. Slack and other similar comm tooling has always been advertised as a productivity booster due to their 'async' nature. Nobody actually believes this anymore and coupling it with the oncall notifications really closes the lid on that thing.

Yeah, unfortunately, I don't think these messaging tools are async. During oncall, I used to pretty much live on Slack. Incidents were on slack, customer tickets on slack, debugging on slack...

That is correct, they are not. My former workplace had Pagerduty integrated with Slack, so I get it...

Re: Show HN: I built an open-source tool to make on-call suck less

#22

Post by theodpHN »

What you've come up with looks helpful (and may have other applications as someone else noted), but you know what also makes on-call suck less? Getting paid for it, in $ and/or generous comp time. :-)

https://betterstack.com/community/guides/incident-management...

Also helpful is having management that is responsive to bad on-call situations and recognizes when capable, full-time around-the-clock staffing is really needed. It seems too few well-paid tech VPs understand what a 7-Eleven management trainee does, i.e., you shouldn't rely on 1st shift workers to handle all the problems that pop up on 2nd and 3rd shift!

Re: Show HN: I built an open-source tool to make on-call suck less

#23

Post by satyamkapoor »

FooBarWidget wrote: post #15
RadiozRadioz wrote: post #8

> Slack-native since that has become the de-facto tool for on-call engineers. In your particular organization. Slack is one of many instant messaging platforms. Tightly coupling your tool to Slack instead of making it platform agnostic immediately restricts where it can be used. Other comment threads are already discussing the broader issues with using IM for this job, so I won't go into it here. Regardless, well don…

Try Netherlands. We're Microsoft land over here. Pretty much everyone is on Azure and Teams. It's mostly startups and hip small companies that use Slack.

Startups, hip small companies, tech product based companies. Most non tech product based or enterprise banks in NL are on Teams

Re: Show HN: I built an open-source tool to make on-call suck less

#24

Post by aray07 »

dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

Yeah, thats fair feedback. The main aim was to reduce the alert fatigue for on-call engineers and provide a way to get insight into the alerts at the end of the on-call shift.

This way there is data to make a case that certain alerts are noisy (for various reasons) and we should strive to reduce the time spent dealing with these alerts. Fixing some of them might be as easy as deleting them but for others might need dedicated time working on them.

Re: Show HN: I built an open-source tool to make on-call suck less

#25

Post by whazor »

solatic wrote: post #11

In my current workplace (BigCo), we know exactly what's wrong with our alert system. We get alerts that we can't shut off, because they (legitimately) represent customer downtime, and whose root cause we either can't identify (lack of observability infrastructure) or can't fix (the fix is non-trivial and management won't prioritize). Running on-call well is a culture problem. You need management to prioritize observa…

Or maybe page your managers, such that they can escalate the situation. They will be more aligned on solving the cultural problems if they get waked up too.

Re: Show HN: I built an open-source tool to make on-call suck less

#26

Post by michaelt »

lars_francke wrote: post #18

Shameless question tangential related to the topic. We are based in Europe and have the problem that some of us sometimes just forget we're on call or are afraid that we'll miss OpsGenie notifications. We're desparately looking for a hardware solution. I'd like something similar to the pagers of the past but at least here in Germany they don't really seem to exist anymore. Ideally I'd have a Bluetooth dongle that ale…

A candy bar cell phone, paid for by your employer and handed to whoever is on call. People who don't want it can just forward it to their phone.

Re: Show HN: I built an open-source tool to make on-call suck less

#27

Post by aray07 »

whazor wrote: post #25
solatic wrote: post #11

In my current workplace (BigCo), we know exactly what's wrong with our alert system. We get alerts that we can't shut off, because they (legitimately) represent customer downtime, and whose root cause we either can't identify (lack of observability infrastructure) or can't fix (the fix is non-trivial and management won't prioritize). Running on-call well is a culture problem. You need management to prioritize observa…

Or maybe page your managers, such that they can escalate the situation. They will be more aligned on solving the cultural problems if they get waked up too.

yeah the best managers i worked with used to be on the same on-call rotation such that they would also get paged every time. That helped build empathy and visibility into the situation.

Re: Show HN: I built an open-source tool to make on-call suck less

#28

Post by dennis_jeeves2 »

aray07 wrote: post #12
solatic wrote: post #11

In my current workplace (BigCo), we know exactly what's wrong with our alert system. We get alerts that we can't shut off, because they (legitimately) represent customer downtime, and whose root cause we either can't identify (lack of observability infrastructure) or can't fix (the fix is non-trivial and management won't prioritize). Running on-call well is a culture problem. You need management to prioritize observa…

I completely agree that technical tools cannot fix culture problems. However, one of the things that I noticed in my previous companies was that my management chain wasn't even aware that the problem was this bad. We also wanted to add better reporting (like the alert analytics) so that people have more visibility into the state of alerts + on-call load on engineers. What strategies have worked well for you when it c…

>However, one of the things that I noticed in my previous companies was that my management chain wasn't even aware that the problem was this bad.

Isn't that a cultural problem?

Re: Show HN: I built an open-source tool to make on-call suck less

#30

Post by crawfishphase »

michaelt wrote: post #26
lars_francke wrote: post #18

Shameless question tangential related to the topic. We are based in Europe and have the problem that some of us sometimes just forget we're on call or are afraid that we'll miss OpsGenie notifications. We're desparately looking for a hardware solution. I'd like something similar to the pagers of the past but at least here in Germany they don't really seem to exist anymore. Ideally I'd have a Bluetooth dongle that ale…

A candy bar cell phone, paid for by your employer and handed to whoever is on call. People who don't want it can just forward it to their phone.

in this case a satellite enabled candybar. the disaster recovery policy and budget should be applicable here. make sure its able to share xg and satellite tunnel for maximum value. ensure the reporting system is satellite enabled also. added points if its sending alerts 2 your handy byod. Disaster recovery is a big deal in 2024. All sorts of factors make satellite redundancy valuable in todays reality: Coworkers on a hike or a boat, random 0-day stuff, and war can cut your normal internet.. i have experienced all of these and only in the last 4 years and more than 1 time on each topic. Train your users to destroy it in case of war as its trackable by military tech. Put a sticker on it. Check out stackexchange for questions like this tho?
Post reply on HN