Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Leveraging AI for efficient incident response

engineering.fb.com

Post reply on HN
Print view View original

21–30 of 58 posts

Re: Leveraging AI for efficient incident response

#21

Post by Kirth »

AeZ1E wrote: post #19

nice to see meta investing in AI investigation tools! but 42% accuracy doesn't sound too impressive to me... maybe there's still some fine-tuning needed for better results? glad to hear about the progress though!

Really, a tool where 42% of incident responses the on call engineers are greeted by a pointer that likely lets them resolve the incident almost immediately and move on, rather than spending potentially hours figuring out which component it is they need to address and how, isn't impressive to you?

Re: Leveraging AI for efficient incident response

#22

Post by nevon »

LASR wrote: post #11

We've shifted our oncall incident response over to mostly AI at this point. And it works quite well. One of the main reasons why this works well is because we feed the models our incident playbooks and response knowledge bases. These playbooks are very carefully written and maintained by people. The current generation of models are pretty much post-human in following them, performing reasoning and suggesting mitigati…

I'm really curious to hear more about what kind of thing is covered in your playbooks. I've often heard and read about the value of playbooks, but I've yet to see it bear fruit in practice. My main work these past few years has been in platform engineering, and so I've also been involved in quite a few incidents over that time, and the only standardized action I can think of that has been relevant over that time is comparing SLIs between application versions and rolling back to a previous version if the newer version is failing. Beyond that, it's always been some new failure mode where the resolution wouldn't have been documented because it's never happened before.

On the investigation side of things I can definitely see how an AI driven troubleshooting process could be valuable. Lots of developers are lacking debugging skills, so an AI driven process that looks at the relevant metrics and logs and can reason around what the next line of inquiry should be could definitely speed things up.

Re: Leveraging AI for efficient incident response

#23

Post by Kiro »

minkles wrote: post #16
donavanm wrote: post #15

Earlier quoted context omitted.

No. Youre missing the UX forest for the pedantry trees here. Ive worked on a team that did similar change detection with little to no ML magic. It matters how its presented as a hint (“top five suggested”) and not THE ANSWER. In addition its VERY common to do things like present confidence or weight to the user. And why theres a huge need for explainability. And this is just part of the diagnosis process. The system…

No I'm not. It's crap. I have about 30 years experience both on hard engineering (electronics) and software engineering particularly on failure analysis and reliability engineering. Most people are lazy and get led astray with false information. This is a very dangerous thing. You need a proper conceptualisation framework like a KT problem analysis to eliminate incorrect causes and keep people thinking rationally and…

Sounds like you're projecting your own laziness and shortcomings on others. This is a tool that seems really helpful considering the alternative is 0%.

Re: Leveraging AI for efficient incident response

#25

Post by chaoz__ »

Kirth wrote: post #21
AeZ1E wrote: post #19

nice to see meta investing in AI investigation tools! but 42% accuracy doesn't sound too impressive to me... maybe there's still some fine-tuning needed for better results? glad to hear about the progress though!

Really, a tool where 42% of incident responses the on call engineers are greeted by a pointer that likely lets them resolve the incident almost immediately and move on, rather than spending potentially hours figuring out which component it is they need to address and how, isn't impressive to you?

It depends on whether it's generating 58% of answers that lead on-call engineers down the wrong path. Honestly, it's more of a question -- I did not read the article deeply.

Re: Leveraging AI for efficient incident response

#26

Post by viraptor »

benreesman wrote: post #17
viraptor wrote: post #12

Earlier quoted context omitted.

You mean it doesn't understand the change you've made based on the diff?

Even under the most generous nomenclature, no contemporary LLM understands anything. They approximate argmax(P_sub_theta(token|prefix)). This approximation is sometimes useful. I’ve found it to never be useful in writing code or prose about code of any difficulty. That’s my personal anecdote, but one will note that OpenAI and Anthropic still employ a great many software engineers.

I know that, likely everyone here knows that. But understanding is a good approximation for what we mean. Pointing out implementation is needlessly pedantic.

Re: Leveraging AI for efficient incident response

#27

Post by BodyCulture »

nyellin wrote: post #3

We've open sourced something with similar goals that you can use today: https://github.com/robusta-dev/holmesgpt/ We're taking a slightly different angle than what Facebook published, in that we're primarily using tool calling and observability data to run investigations. What we've released really shines at surfacing up relevant observability data automatically, and we're soon planning to add the change-tracking ele…

Can we see the recording of this webinar somewhere?

Re: Leveraging AI for efficient incident response

#28

Post by BodyCulture »

nyellin wrote: post #3

We've open sourced something with similar goals that you can use today: https://github.com/robusta-dev/holmesgpt/ We're taking a slightly different angle than what Facebook published, in that we're primarily using tool calling and observability data to run investigations. What we've released really shines at surfacing up relevant observability data automatically, and we're soon planning to add the change-tracking ele…

https://news.ycombinator.com/item?id=41327430

Re: Leveraging AI for efficient incident response

#29

Post by UK-AL »

re-thc wrote: post #20
donavanm wrote: post #4

Im really interested in the implied restriction/focus on “code changes.” IME a very very large number of impacting incidents arent strictly tied to “a” code change, if any at all. It _feels_ like theres an implied solution to tying running version back to deployment rev, to deployment artifacts, and vcs. Boundary conditions and state changes in the distributed system were the biggest bug bear I ran in to at AWS. Then…

> IME a very very large number of impacting incidents arent strictly tied to “a” code change, if any at all Usually this implies there are bigger problems. If something keeps breaking without any change (config / code) then it was likely always broken and just ignored. So when companies do have most of the low hanging fruit resolved it's the changes that break things. I've seen places where everything is duck taped t…

At my place 90% of them are 3rd parties going down, and you can't do much other than leave. But the new 3rd parties are just as bad. All you can do gracefully handle failure.

Re: Leveraging AI for efficient incident response

#30

Post by TheBengaluruGuy »

Interesting. Just a few weeks back, I was reading about their previous work https://atscaleconference.com/the-evolution-of-aiops-at-meta... -- didn't realise there's more work!

Also, some more researches in the similar space by other enterprises:

Microsoft: https://yinfangchen.github.io/assets/pdf/rcacopilot_paper.pd...

Salesforce: https://blog.salesforceairesearch.com/pyrca/

Personal plug: I'm building a self-service AIOps platform for engineering teams (somewhat similar to this work by Meta). If you're looking to read more about it, visit -- https://docs.drdroid.io/docs/doctor-droid-aiops-platform

Post reply on HN