Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built an open-source tool to make on-call suck less

github.com

Post reply on HN
Print view View original

61–70 of 174 posts

Re: Show HN: I built an open-source tool to make on-call suck less

#61

Post by mike_d »

__turbobrew__ wrote: post #19
solatic wrote: post #11

In my current workplace (BigCo), we know exactly what's wrong with our alert system. We get alerts that we can't shut off, because they (legitimately) represent customer downtime, and whose root cause we either can't identify (lack of observability infrastructure) or can't fix (the fix is non-trivial and management won't prioritize). Running on-call well is a culture problem. You need management to prioritize observa…

I work on a team which runs hyper critical infra on all production machines at BigCo and have the same experience as you. The problem are not the alerts — the alerts actually are catching real problems — the problem is the following: 1. The team is understaffed so sometimes spending a few days root causing an alert is not prioritized 2. When alerts are root caused sometimes the work to fix the root cause is not prior…

Start putting together conference bridges for "P1 customer outages" and have someone who is responsible for calling the developers, PMs, scrum masters, managers, etc. on the team and getting them all on at 1 AM to fix it.

Re: Show HN: I built an open-source tool to make on-call suck less

#62

Post by hackeman300 »

lupire wrote: post #60
anitil wrote: post #51

Earlier quoted context omitted.

I have half-jokingly suggested that an out-of-hours page should cost the company $10k to incentivise actually fixing problems rather than releasing broken products. But I haven't thought of a way of getting around the perverse incentive to create bugs in order to get the $10k

The cost is the cost of paying you to fix outages on overtime pay instead of working on the product.

Overtime pay? Is this common for oncalls? Never gotten it myself, every time I ask they reply with "just take the time back on another day" as if my time is fungible. Weekend time is worth far more to me than weekday time

Re: Show HN: I built an open-source tool to make on-call suck less

#63

Post by [deleted] »

dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

[deleted]

Re: Show HN: I built an open-source tool to make on-call suck less

#64

Post by Aurornis »

anitil wrote: post #51
whazor wrote: post #25

Earlier quoted context omitted.

Or maybe page your managers, such that they can escalate the situation. They will be more aligned on solving the cultural problems if they get waked up too.

I have half-jokingly suggested that an out-of-hours page should cost the company $10k to incentivise actually fixing problems rather than releasing broken products. But I haven't thought of a way of getting around the perverse incentive to create bugs in order to get the $10k

One company I worked for introduced a trivial ($100 or something) gift card bonus for closing a certain number of bug tickets.

The number of people who started pushing code with subtle bugs so they could create a ticket for it, fix their own bug, and get closer to that $100 gift card was shocking to me.

I can’t imagine the chaos that would occur if something came with a $10K bonus attached. Some people will bend over backward to get even tiny rewards. Dangling a $10K reward would get the wheels turning in their heads immediately.

Re: Show HN: I built an open-source tool to make on-call suck less

#65

Post by Aurornis »

hackeman300 wrote: post #62
lupire wrote: post #60

Earlier quoted context omitted.

The cost is the cost of paying you to fix outages on overtime pay instead of working on the product.

Overtime pay? Is this common for oncalls? Never gotten it myself, every time I ask they reply with "just take the time back on another day" as if my time is fungible. Weekend time is worth far more to me than weekday time

Some companies do on-call bonuses, overtime pay for on-call incidents, or other schemes.

In my experience, it’s not a net win. They’ve budgeted the same amount for compensation either way, so you’re probably getting lower base comp if they’re allocating some of it for on-call.

It also creates an atmosphere where on-call becomes more normalized, because you’re getting paid extra to do it. Some people, usually young single people, will try to milk the overtime for as much as they can, dragging out the hours spent doing on-call work because every extra hour spent on the problem makes their paycheck bigger.

Re: Show HN: I built an open-source tool to make on-call suck less

#66

Post by [deleted] »

linuxdude314 wrote: post #52
lars_francke wrote: post #18

Shameless question tangential related to the topic. We are based in Europe and have the problem that some of us sometimes just forget we're on call or are afraid that we'll miss OpsGenie notifications. We're desparately looking for a hardware solution. I'd like something similar to the pagers of the past but at least here in Germany they don't really seem to exist anymore. Ideally I'd have a Bluetooth dongle that ale…

This really sounds like a _you_ problem and not something you need hardware to fix. You can already enable silence/focus time bypass modes for apps like PagerDuty and such… If you can’t develop some sense of responsibility to check if you’re on-call, frankly you have no business being in an on-call role. No hardware will make you or your engineers more diligent. The only reason pagers made more sense than phones is/w…

[deleted]

Re: Show HN: I built an open-source tool to make on-call suck less

#67

Post by DanHulton »

dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

I agree with your intent and desire, but the fact is that this problem _keeps happening,_ and we can't fix it by advocating "well, just do alerts better." There's a lot of cultural inertia at a lot of places that leads to creating too many, too low-signal alerts, and fixing that across an entire company -- or, hell, industry -- is a magnificently tall order.

However, installing a tool to specifically reign in those garbage noisy alerts is a potentially easy, significant win for the time and mental health of on-call engineers.

I mean, it sounds like you can then afterwards go in and identify the alerts that are just noise, and having that data means you can take action. Maybe contact the teams that are writing the noisiest alerts, or prepare some data-driven engineering standards for the company, whatever. But that still falls into "fix the culture", which is famously hard to do by fiat.

Re: Show HN: I built an open-source tool to make on-call suck less

#68

Post by stitched2gethr »

dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

If I may try to counter this point.

You are right, but there's ideal state and there's the real world. When on call most of my time is spent trying to make on call better. Reducing noise, providing more context when the alert and logs are lacking, and of course fixing the real issues that alerts have identified. That said there is a period of time in between receiving non-actionable alerts and classifying them as such, and more context without using brain power is always welcome. I think I'll give it a shot.

Re: Show HN: I built an open-source tool to make on-call suck less

#69

Post by Geezus_42 »

jobtemp wrote: post #39
lars_francke wrote: post #18

Shameless question tangential related to the topic. We are based in Europe and have the problem that some of us sometimes just forget we're on call or are afraid that we'll miss OpsGenie notifications. We're desparately looking for a hardware solution. I'd like something similar to the pagers of the past but at least here in Germany they don't really seem to exist anymore. Ideally I'd have a Bluetooth dongle that ale…

There are phone apps that can pierce through all silent or DND settings. Get one of those. If the same app could buzz on less than 50% battery to remind to charge that would help. Also same app could request to confirm on call status so the don't forget. If they don't confirm someone else gets the shift.

OpsGenie has that. We use it at my job. I'm not sure what problem OP is having. The phone call, text message, and app alert from OpsGenie are more than enough. The notification configuration is extremely flexible and each user can customize it as needed. From a user perspective, I don't know what else you could want.

I have no affiliation with OpsGenie outside of using at work.

Re: Show HN: I built an open-source tool to make on-call suck less

#70

Post by october8140 »

RadiozRadioz wrote: post #8

> Slack-native since that has become the de-facto tool for on-call engineers. In your particular organization. Slack is one of many instant messaging platforms. Tightly coupling your tool to Slack instead of making it platform agnostic immediately restricts where it can be used. Other comment threads are already discussing the broader issues with using IM for this job, so I won't go into it here. Regardless, well don…

As Slack is not end to end encrypted we and I imagine many other companies cannot use it.
Post reply on HN