Show HN: I built an open-source tool to make on-call suck less • Hacker News Board

Re: Show HN: I built an open-source tool to make on-call suck less

#171

Post by yreg » Tue, Jul 30, 2024, 2:53 PM UTC

Flop7331 wrote: post #168

yreg wrote: post #165

Earlier quoted context omitted.

I'm not sure if you are trolling or genuine, but obviously it is worth it to wake someone up (someone who is specifically paid for being available to be woken up) once it prevents enough costs by resolving the issue now instead of doing so in half a day.

> I'm not sure if you are trolling or genuine Odd, because I'm not sure if you are either. Very few things are so important or costly, and if you're winding up in that situation frequently enough to rob people of their personal time to manually deal with it, clearly something is majorly wrong with this hypothetical critical thing at an architectural level. There's nothing controversial about this.

> Very few things are so important or costly

Any outage that costs more than paying an engineer to be on-call is worth it. It's not that complex. If an outage blocks 10000 people from doing their work, it's obviously worth it to wake someone up to try to resolve it half a day sooner. (Someone you've been paying specifically for this purpose!)

> rob people of their personal time

Being paid to be on-call is not your personal time.

Re: Show HN: I built an open-source tool to make on-call suck less

#172

Post by esafak » Tue, Jul 30, 2024, 9:43 PM UTC

Flop7331 wrote: post #112

kgeist wrote: post #105

Earlier quoted context omitted.

System goes down or degrades in some other way at night and important customers with a different timezone get angry, threatening to leave? (happened with us a few times) But I would'nt use LLM for it due to hallucinations

What's so important that your customers in other timezones feel like waking you up? If they're ready to walk that fast, you can't trust them not to ditch you for an alternative as soon as they find one.

You don't trust them to ditch you. That's what agreements are for. And your track record is part of what makes them sign and extend the agreement.

Re: Show HN: I built an open-source tool to make on-call suck less

#173

Post by Flop7331 » Tue, Jul 30, 2024, 10:02 PM UTC

yreg wrote: post #171

Flop7331 wrote: post #168

Earlier quoted context omitted.

> I'm not sure if you are trolling or genuine Odd, because I'm not sure if you are either. Very few things are so important or costly, and if you're winding up in that situation frequently enough to rob people of their personal time to manually deal with it, clearly something is majorly wrong with this hypothetical critical thing at an architectural level. There's nothing controversial about this.

> Very few things are so important or costly Any outage that costs more than paying an engineer to be on-call is worth it. It's not that complex. If an outage blocks 10000 people from doing their work, it's obviously worth it to wake someone up to try to resolve it half a day sooner. (Someone you've been paying specifically for this purpose!) > rob people of their personal time Being paid to be on-call is not your pe…

Ah, it's a difference in lived experiences. I was certainly never compensated for it when I had to do it.

Re: Show HN: I built an open-source tool to make on-call suck less

#174

Post by CableNinja » Thu, Aug 01, 2024, 4:25 PM UTC

I get your sentiment, but theres another side of this coin that everyone is forgetting, hilariously.

You can tune your monitoring!

Noisy alert that tends to be a false positive but not always? Tune alert message to only send if the issue continues for more than a minute, or if the check fails 3 times in a row. Theres hundreds of ways to tweak a monitor to match your environment.

Best of all? It takes 30 seconds at most. Find the trigger, adjust slightly, and after maybe 1-2 tries, youll be getting 1 false positive sometimes, and actual alerts when they happen, compared to 99% false alerts, all the time.

Oh and did you know any monitoring solution worth its salt can execute things automatically on alerts, and then can alert you if that thing fails?

Also, Slack is not a defacto anything. Its a chat tool in a world of chat tools