Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built an open-source tool to make on-call suck less

github.com

Post reply on HN
Print view View original

151–160 of 174 posts

Re: Show HN: I built an open-source tool to make on-call suck less

#151

Post by henryfjordan »

cdchn wrote: post #142
henryfjordan wrote: post #136

Earlier quoted context omitted.

Depending on the stakes this is a pretty dangerous attitude. The goal for oncall is to keep the website working, and if you're tuning for "never get paged" then you'll necessarily miss an incident eventually.

If you make your goal as high availability as possible, and you only get paged on outages, then your goal should be to never get paged. You should be building resilient architectures, not being on firewatch duty.

This is a classic developer vs business incentives misalignment.

Developers don't want to ever be paged because they don't want to be bothered, but the business might be perfectly happy to pay you to be on firewatch duty.

Consider a "low traffic" alert, how can you tell the difference between a slow period at 3am on a holiday vs a true outage? You can't without someone getting up and testing if the site is still up. (Maybe you can automate that check but there's always edge-cases you can't automate).

OP seemed to suggest it's better to disable the alarm than to just suffer the false alarm every now and then. I doubt very much that the people paying you for the on-call service would agree though.

Re: Show HN: I built an open-source tool to make on-call suck less

#152

Post by makmanalp »

Underrated oncall problem that needs solving is scheduling IMHO:

- We have a weekday (2 shifts) / weekend (1 slightly longer shift including friday morning to allow people to take long weekends) oncall rotation as well as a group-combined oncall schedule which gets finnicky.

- When people join or leave the rotation, making sure nothing shifts before a certain date or swapping one person with another without changing the rest and other things are a massive pain in the butt

- Combine this with a company holiday list - usually there's different policies and expectations during those. - Allow custom shift change times for people in different timezones.

- We have "oncall training" / shadowing for newbies, automate the process of substituting them in gradually, first with a shared daytime rotation and then on their own etc.

- Make oncall trades (if you can't make your shift simpler)

Gripes with PD:

- Pagerduty keeps insisting I'm "always on call" because I'm on level N of a fallback pager chain which makes their "when oncall next" box useless - just let me pick.

- Similarly, pagerduty's google calendar export will just jam in every service you're remotely related to and won't let you pick when exporting, even though it will in their UI. So I can't just have my oncall schedule in google calendar without polluting it to all hell.

Re: Show HN: I built an open-source tool to make on-call suck less

#153

Post by aflag »

jofer wrote: post #125
aflag wrote: post #108

It feels to me that using LLM to classify alerts as noisy is just adding risk instead of fixing the root cause of the problem. If an alert is known to be noisy and have appeared on slack before (which is how the LLM would figure out it's a noisy alert), then just remove the alert? Otherwise, how will the LLM know it's noise? Either it will correctly annoy you or hallucinate a reason it figures that alert is just nois…

There is a lot to be said for "smoke test" metrics. Things you expect to have frequent false positives, but are sometimes early indicators of larger problems or indicators of where to look deeper if something else goes sideways. They're not things that should wake you up in the middle of the night, but they're a damn valuable tool to quickly figure out what's actually wrong when a "real" alert triggers. Many of these…

That's why alerts can have different priority levels. So, less serious issues can be addressed during normal working hours (eg. disk is 80% full). Maybe LLMs will figure out the correct P level for something like disk usage, but it's unlikely to get it right for things that are particular to your application. Maybe use an LLM when you're creating the alert to auto fill the priority level. That can then be verified by someone. Don't silence an alert based on what an LLM thinks though.

Re: Show HN: I built an open-source tool to make on-call suck less

#154

Post by jpb0104 »

I love this space; stability & response! After my last full-time gig, I was also frustrated with the available tooling and ONLY wanted an on-call scheduling tool with simple calendar integration. So I built: https://majorpager.com/ Not OSS, but very simple and hopefully pretty straightforward to use. I'm certainly wide open to feedback.

Re: Show HN: I built an open-source tool to make on-call suck less

#155

Post by aray07 »

jedberg wrote: post #139

People do not understand the value of classifying alerts as useful after the fact . At Netflix we built a feature into our alert systems that added a simple button at the top of every alert that said, "Was this alert useful?". Then we would send the alert owners reports about what percent of people found their alert useful. It really let us narrow in on which alerts were most useful so that others could subscribe the…

Yeah, that was one of the goals we had. We try to classify when an alert comes up and let the engineer give us feedback.

We use that to generate a report so that teams have visibility into which alerts are causing the most amount of noise.

Re: Show HN: I built an open-source tool to make on-call suck less

#156

Post by aray07 »

williamdclt wrote: post #133
dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

> which upon investigation might not even be “just” noise My company (like so many) is struggling a bit with culture around noisy alarm. Not only is noise tolerated, but when some closes an alarm because it's "known to be noise" and I prod them, it turns out that there is a very real impact on the user, it's just that nobody bothered to look into it. The alarm rings, the on-call hopes that it closes itself soon enoug…

Yeah, I was surprised at how common this behavior has been across companies. I don't have a good solution for this problem - I am hoping additional visibility and accountability into these alerts can be useful. One of the things we have been hoping to do is to be able to add user impact context into the alert enrichment as well.

Re: Show HN: I built an open-source tool to make on-call suck less

#157

Post by aray07 »

makmanalp wrote: post #152

Underrated oncall problem that needs solving is scheduling IMHO: - We have a weekday (2 shifts) / weekend (1 slightly longer shift including friday morning to allow people to take long weekends) oncall rotation as well as a group-combined oncall schedule which gets finnicky. - When people join or leave the rotation, making sure nothing shifts before a certain date or swapping one person with another without changing…

Thanks for the feedback! I completely relate to PD scheduling issues and something that we want to take a look at as well.

Re: Show HN: I built an open-source tool to make on-call suck less

#158

Post by Jolter »

jedberg wrote: post #139

People do not understand the value of classifying alerts as useful after the fact . At Netflix we built a feature into our alert systems that added a simple button at the top of every alert that said, "Was this alert useful?". Then we would send the alert owners reports about what percent of people found their alert useful. It really let us narrow in on which alerts were most useful so that others could subscribe the…

If each and every alert has an owner, you’ve solved half of the cultural problem already. Good on you!

Re: Show HN: I built an open-source tool to make on-call suck less

#159

Post by Jolter »

ravedave5 wrote: post #132

The goal for oncall should be to NEVER get called. If someone gets called when they are oncall their #1 task the next day is to make sure that call never happens again. That means either fixing a false alarm or tracking down the root cause of the call. Eventually you get to a state where being called is by far the exception instead of the norm.

This is how my team used to work when I was on call in telecoms a decade ago. In the right engineering culture, and with management buy-in, it works really well.

We deployed a new system and had one week on call for each of five team members. The first couple of rotations were hell. Almost every night ended up with at least one wake up call. As we learned how to solve each type of outage, we then taught the first-line staff how to reboot the right components so we didn’t get as many wake-ups, while we spent our days fixing the bugs. And eventually the system stopped crashing.

The on-call pay was really good (nearly double for that week) and it was a pretty sweet reward to be able to rake that in as calls stopped coming. We broke out a bottle of champagne when the first week of no calls had passed.

Eventually on-call was cancelled.

Imagine how this story would have ended if management had incentivized us differently, for example if you only got the extra pay for the nights where you got pages.

Re: Show HN: I built an open-source tool to make on-call suck less

#160

Post by bckr »

henryfjordan wrote: post #151
cdchn wrote: post #142

Earlier quoted context omitted.

If you make your goal as high availability as possible, and you only get paged on outages, then your goal should be to never get paged. You should be building resilient architectures, not being on firewatch duty.

This is a classic developer vs business incentives misalignment. Developers don't want to ever be paged because they don't want to be bothered, but the business might be perfectly happy to pay you to be on firewatch duty. Consider a "low traffic" alert, how can you tell the difference between a slow period at 3am on a holiday vs a true outage? You can't without someone getting up and testing if the site is still up.…

> Developers don't want to ever be paged because they don't want to be bothered

This is a very reductive statement.

Developers have experienced their best colleagues burning out and leaving jobs because of on-call being completely overwhelming.

Developers want to behave intelligently.

Developers want the system to work.

Developers don’t want to burn their lifespan for false alarms that are being sent because someone didn’t spend 30 seconds thinking about whether a human being needs to be woken up in the middle of the night for whatever widget they’re slapping together.

Post reply on HN