Live data from Hacker News

Show HN: I built an open-source tool to make on-call suck less

github.com

31–40 of 174 posts

Re: Show HN: I built an open-source tool to make on-call suck less

#31
Using LLMs to classify noisy alerts is a really clever approach to tackling alert fatigue! Are you fine tuning your own model to differentiate between actionable and noisy alerts?

I'm also working on an open source incident management platform called Incidental (https://github.com/incidentalhq/incidental), slightly orthogonal to what you're doing, and it's great to see others addressing these on-call challenges.

Our tech stacks are quite similar too - I'm also using Python 3, FastAPI!

Re: Show HN: I built an open-source tool to make on-call suck less

#32

Nice job and congratulations on building this! It looks like your copy is missing a word in the first paragraph: > Opslane is a tool that helps (make) the on-call experience less stressful.

derp, thanks for catching. It has been fixed!

Re: Show HN: I built an open-source tool to make on-call suck less

#33
post #25
post #11

In my current workplace (BigCo), we know exactly what's wrong with our alert system. We get alerts that we can't shut off, because they (legitimately) represent customer downtime, and whose root cause we either can't identify (lack of observability infrastructure) or can't fix (the fix is non-trivial and management won't prioritize). Running on-call well is a culture problem. You need management to prioritize observa…

Or maybe page your managers, such that they can escalate the situation. They will be more aligned on solving the cultural problems if they get waked up too.

Or maybe page your managers, such that they can fire you

Re: Show HN: I built an open-source tool to make on-call suck less

#34
post #33
post #25

Earlier quoted context omitted.

Or maybe page your managers, such that they can escalate the situation. They will be more aligned on solving the cultural problems if they get waked up too.

Or maybe page your managers, such that they can fire you

Then... problem solved!

Re: Show HN: I built an open-source tool to make on-call suck less

#35
post #31

Using LLMs to classify noisy alerts is a really clever approach to tackling alert fatigue! Are you fine tuning your own model to differentiate between actionable and noisy alerts? I'm also working on an open source incident management platform called Incidental ( https://github.com/incidentalhq/incidental ), slightly orthogonal to what you're doing, and it's great to see others addressing these on-call challenges. Ou…

Why not use statistics? Been reading about xmr charts recently on commoncog. That might help for example.

Re: Show HN: I built an open-source tool to make on-call suck less

#36
One of the “no-bullshit” positions I have arrived at over the years is that “real-time is a gimmick”.

You don’t need that Times Square ad, only 8-10 people will look up. If you just want the footage of your conspicuous consumotion, you can easily photoshop it for decades already.

Similarly, chat causes anxiety and lack of productivity. Threaded forums like HN are better. Having a system to prevent problems and the rare emergency is better than having everyone glued to their phones 24/7. And frankly, threads keep information better localized AND give people a chance to THINK about the response and iterate before posting in a hurry. When producers of content take their time, this creates efficiencies for EVERY INTERACTION WITH that content later, and effects downstream. (eg my caps lock gaffe above, I wont go back and fix it, will jjst keesp typing 111!1!!!)

Anyway people, so now we come to today’s culture. Growing up I had people call and wish happy birthday. Then they posted it on FB. Then FB automated the wishes so you just press a button. Then people automated the thanks by pressing likes. And you can probably make a bot to automate that. What once was a thoughtful gesture has become commoditized with bots talking to bots.

Similar things occurred with resumes and job applications etc.

So I say, you want to know my feedback? Add an AI agent that replies back with basic assurances and questions to whoever “summoned you”, have the AI fill out a form, and send you that. The equivalent of front-line call center workers asking “Have you tried turning it on and off again” and “I understand it doesn’t work, but how can we replicate it.”

That repetitive stuff should he done by AI and build up an FAQ Knowledge Base for bozos and then only bother you if it came across a novel problem it hasn’t solved yet, like an emergency because, say, there’s a windows BSOD spreading and systems don’t boot up. Make the AI do triage and tell the differencd.

Re: Show HN: I built an open-source tool to make on-call suck less

#37
post #11

In my current workplace (BigCo), we know exactly what's wrong with our alert system. We get alerts that we can't shut off, because they (legitimately) represent customer downtime, and whose root cause we either can't identify (lack of observability infrastructure) or can't fix (the fix is non-trivial and management won't prioritize). Running on-call well is a culture problem. You need management to prioritize observa…

> Running on-call well is a culture problem. You need management to prioritize observability (you can't fix what you can't show as being broken), then you need management to build a no-broken-windows culture (feature development stops if anything is broken).

I was lucky enough to join a company where management does this. The managers were made to do this by experienced engineers who explained to them in no uncertain terms that stuff was broken and nothing was being shipped until things stopped being broken. Unless you have good managers this won’t happen without a fight and it’s a fight I think we as engineers need to take.

Some managers in other teams played the “oh it’s not super high impact it’s not prioritized” game, and those teams now own a bunch of broken stuff and make very slow progress because their developers are tiptoeing around broken glass, and end up building even more broken stuff because nothing they own is robust. Those managers played themselves.

Communication with management is bidirectional, sometimes they need a lot of persuasion.

Re: Show HN: I built an open-source tool to make on-call suck less

#38
post #31

Using LLMs to classify noisy alerts is a really clever approach to tackling alert fatigue! Are you fine tuning your own model to differentiate between actionable and noisy alerts? I'm also working on an open source incident management platform called Incidental ( https://github.com/incidentalhq/incidental ), slightly orthogonal to what you're doing, and it's great to see others addressing these on-call challenges. Ou…

Thanks for the feedback! I saw the incidental launch on HN and have been following your journey!

Re: Show HN: I built an open-source tool to make on-call suck less

#39

Shameless question tangential related to the topic. We are based in Europe and have the problem that some of us sometimes just forget we're on call or are afraid that we'll miss OpsGenie notifications. We're desparately looking for a hardware solution. I'd like something similar to the pagers of the past but at least here in Germany they don't really seem to exist anymore. Ideally I'd have a Bluetooth dongle that ale…

There are phone apps that can pierce through all silent or DND settings. Get one of those. If the same app could buzz on less than 50% battery to remind to charge that would help. Also same app could request to confirm on call status so the don't forget. If they don't confirm someone else gets the shift.

Re: Show HN: I built an open-source tool to make on-call suck less

#40
post #11

In my current workplace (BigCo), we know exactly what's wrong with our alert system. We get alerts that we can't shut off, because they (legitimately) represent customer downtime, and whose root cause we either can't identify (lack of observability infrastructure) or can't fix (the fix is non-trivial and management won't prioritize). Running on-call well is a culture problem. You need management to prioritize observa…

That’s true. But technical tools can help you highlight culture problems so that they’re easier to to discuss and fix. It’s been a minute since I’ve had to process exactly the kind of on-call/alert problem we’re discussing here, but this does feel like the kind of tool that would help sell the kinds of management/culture changes necessary to really improve things, if not fix all of them.

Switching tools, or adopting new (unproven) ones doesn't address or fix the communication issue.

The existing tools mentioned can show the metrics. Management needs an education - and that is part of the engineering job.

Post reply on HN