Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built an open-source tool to make on-call suck less

github.com

Post reply on HN
Print view View original

161–170 of 174 posts

Re: Show HN: I built an open-source tool to make on-call suck less

#161

Post by jofer »

aflag wrote: post #153
jofer wrote: post #125

Earlier quoted context omitted.

There is a lot to be said for "smoke test" metrics. Things you expect to have frequent false positives, but are sometimes early indicators of larger problems or indicators of where to look deeper if something else goes sideways. They're not things that should wake you up in the middle of the night, but they're a damn valuable tool to quickly figure out what's actually wrong when a "real" alert triggers. Many of these…

That's why alerts can have different priority levels. So, less serious issues can be addressed during normal working hours (eg. disk is 80% full). Maybe LLMs will figure out the correct P level for something like disk usage, but it's unlikely to get it right for things that are particular to your application. Maybe use an LLM when you're creating the alert to auto fill the priority level. That can then be verified by…

Yeah, I completely agree. I just meant that alerts you don't immediately respond to or are "noisy" aren't necessarily things you want to delete. Having low priority "noisy" alerts is not a bad thing.

Re: Show HN: I built an open-source tool to make on-call suck less

#162

Post by henryfjordan »

bckr wrote: post #160
henryfjordan wrote: post #151

Earlier quoted context omitted.

This is a classic developer vs business incentives misalignment. Developers don't want to ever be paged because they don't want to be bothered, but the business might be perfectly happy to pay you to be on firewatch duty. Consider a "low traffic" alert, how can you tell the difference between a slow period at 3am on a holiday vs a true outage? You can't without someone getting up and testing if the site is still up.…

> Developers don't want to ever be paged because they don't want to be bothered This is a very reductive statement. Developers have experienced their best colleagues burning out and leaving jobs because of on-call being completely overwhelming. Developers want to behave intelligently. Developers want the system to work. Developers don’t want to burn their lifespan for false alarms that are being sent because someone…

> The goal for oncall should be to NEVER get called.

Is that not also reductive then? Or maybe my statement pretty accurately captures that sentiment without 4 sentences of explanation.

But no, instead of engaging with the meat of my argument you just reductively attack one sentence.

I get it, I'm oncall right now for my job. I don't like it when alarms go off. I also understand that if I were to tune the alarms so I "NEVER get called" I'd be out of a job soon enough because the business would go under.

Re: Show HN: I built an open-source tool to make on-call suck less

#163

Post by asdf6969 »

I don’t really understand the use case. If there’s a way to programmatically tell that it’s a false alarm then there must also be a way to not create the alert in the first place

I’ve never seen an issue that’s conclusively a false alarm without investigating at all. Just delete the alarm? An LLM will never find something like another team is accidentally stress testing my service but it does happen

Another perfect example is when the queen died and it looked like an outage for UK users. Can your LLM read the news? ChatGPT doesn’t even know if she’s alive

I expect you will need AGI before large companies will trust your product.

Re: Show HN: I built an open-source tool to make on-call suck less

#164

Post by cdchn »

henryfjordan wrote: post #151
cdchn wrote: post #142

Earlier quoted context omitted.

If you make your goal as high availability as possible, and you only get paged on outages, then your goal should be to never get paged. You should be building resilient architectures, not being on firewatch duty.

This is a classic developer vs business incentives misalignment. Developers don't want to ever be paged because they don't want to be bothered, but the business might be perfectly happy to pay you to be on firewatch duty. Consider a "low traffic" alert, how can you tell the difference between a slow period at 3am on a holiday vs a true outage? You can't without someone getting up and testing if the site is still up.…

This is classic misalignment of business needs vs. perceived management wants.

Are they paying you to answer false alarms, or are they paying you to make sure the site is available and performant to keep customers happy? Nobody with half a brain wants to answer a bunch of false alarms. Are there are people that will happily get paid to ACK yet another noisy alarm just to collect a paycheck? Certainly; but these are button pushers, not problem solvers.

Your low traffic alert scenario simply requires synthetic requests. This is you you test anything with low usage, but requires high reliability.

Re: Show HN: I built an open-source tool to make on-call suck less

#165

Post by yreg »

Flop7331 wrote: post #117
kgeist wrote: post #114

Earlier quoted context omitted.

Well, I guess it depends on the business. I forgot to mention that we're B2B. For example, suppose a large food chain or a major bank has an important exam scheduled for their employees on a specific day. If our platform has a blocking bug, no one can proceed (some may be sitting in the class) because the developers are too busy sleeping. Some of our clients are also airplane pilot certification authorities, which ha…

But those are the kinds of scenarios where I imagine the sun still came up if comparable disruptions occurred prior to our current era of constant connectivity. We're too invested in the myth that our special problem can't wait half a day.

I'm not sure if you are trolling or genuine, but obviously it is worth it to wake someone up (someone who is specifically paid for being available to be woken up) once it prevents enough costs by resolving the issue now instead of doing so in half a day.

Re: Show HN: I built an open-source tool to make on-call suck less

#166

Post by bckr »

henryfjordan wrote: post #162
bckr wrote: post #160

Earlier quoted context omitted.

> Developers don't want to ever be paged because they don't want to be bothered This is a very reductive statement. Developers have experienced their best colleagues burning out and leaving jobs because of on-call being completely overwhelming. Developers want to behave intelligently. Developers want the system to work. Developers don’t want to burn their lifespan for false alarms that are being sent because someone…

> The goal for oncall should be to NEVER get called. Is that not also reductive then? Or maybe my statement pretty accurately captures that sentiment without 4 sentences of explanation. But no, instead of engaging with the meat of my argument you just reductively attack one sentence. I get it, I'm oncall right now for my job. I don't like it when alarms go off. I also understand that if I were to tune the alarms so I…

Okay, dialing up good-faith engagement.

How would your interpretation change if the article said this instead?

> The goal for oncall should be to continuously tune the system toward having no outages and no false alarms.

FWIW, I did only attack one sentence. This was not exactly intended to be dismissive. It was my reaction to, in my eyes, the weakest part of your argument.

Re: Show HN: I built an open-source tool to make on-call suck less

#167

Post by adamredwoods »

dclowd9901 wrote: post #20

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and…

Agreed. To solve noisy alerts, we have a group channel where we 1. call it out 2. solve it with weekly meetings. We usually "group" up and fix it, so everyone is in agreement. It takes about an hour to organize everyone and talk through it, but it improves the on-call quality-of-life over time.

Re: Show HN: I built an open-source tool to make on-call suck less

#168

Post by Flop7331 »

yreg wrote: post #165
Flop7331 wrote: post #117

Earlier quoted context omitted.

But those are the kinds of scenarios where I imagine the sun still came up if comparable disruptions occurred prior to our current era of constant connectivity. We're too invested in the myth that our special problem can't wait half a day.

I'm not sure if you are trolling or genuine, but obviously it is worth it to wake someone up (someone who is specifically paid for being available to be woken up) once it prevents enough costs by resolving the issue now instead of doing so in half a day.

> I'm not sure if you are trolling or genuine

Odd, because I'm not sure if you are either.

Very few things are so important or costly, and if you're winding up in that situation frequently enough to rob people of their personal time to manually deal with it, clearly something is majorly wrong with this hypothetical critical thing at an architectural level.

There's nothing controversial about this.

Re: Show HN: I built an open-source tool to make on-call suck less

#169

Post by steve1977 »

karamanolev wrote: post #91
steve1977 wrote: post #75

Earlier quoted context omitted.

If you work at a shitty place, focus your energy on leaving the shitty place.

The shittiness of a place is defined by a lot more, and more important, attributes than alert hygiene. Culture, pay, location, industry, leadership. If one leaves companies for (relatively) minor things like that, there are basically no companies left to work for.

Alert hygiene is a symptom of bad culture. Not the only one of course.

Re: Show HN: I built an open-source tool to make on-call suck less

#170

Post by Arch-TK »

sir_eliah wrote: post #124
Arch-TK wrote: post #101

We could stop normalising "on-call" instead.

Could you please elaborate?

Yes, increasingly companies are pretending like their SAAS needs to run with at least 99.999% uptime and so are insisting that all their engineers/programmers/whatevers must therefore be happy to be on-call on a rota for no extra pay because of vagueness in their contracts.

Meanwhile they either have a global workforce so don't actually need to have anyone on-call or only have customers in countries they have employees in.

It's bullshit.

Either companies should be up front about this when hiring or it should be optional and paid.

Or, they can use their engineering talent, just like telecoms companies have been doing for ever, to engineer their products to be more resilient and automate failure cases so proper remediation can wait until working hours.

Post reply on HN