Leveraging AI for efficient incident response
engineering.fb.com
Leveraging AI for efficient incident response
1–10 of 58 posts
Re: Leveraging AI for efficient incident response
#2This is really cool. My optimistic take on GenAI, at least with regard to software engineering, is that it seems like we're gonna have a lot of the boring / tedious parts of our jobs get a lot easier!
Re: Leveraging AI for efficient incident response
#3We've open sourced something with similar goals that you can use today: https://github.com/robusta-dev/holmesgpt/
We're taking a slightly different angle than what Facebook published, in that we're primarily using tool calling and observability data to run investigations.
What we've released really shines at surfacing up relevant observability data automatically, and we're soon planning to add the change-tracking elements mentioned in the Facebook post.
If anyone is curious, I did a webinar with PagerDuty on this recently.
Re: Leveraging AI for efficient incident response
#4Im really interested in the implied restriction/focus on “code changes.”
IME a very very large number of impacting incidents arent strictly tied to “a” code change, if any at all. It _feels_ like theres an implied solution to tying running version back to deployment rev, to deployment artifacts, and vcs.
Boundary conditions and state changes in the distributed system were the biggest bug bear I ran in to at AWS. Then below that were all of the “infra” style failures like network faults, latency, API quota exhaustion, etc. And for all the cloudformation/cdk/terraform in the world its non trivial to really discover those effects and tie them to a “code change.” Totally ignoring older tools that may be managed via CLI or the ol’ point and click.
Re: Leveraging AI for efficient incident response
#5> The biggest lever to achieving 42% accuracy was fine-tuning a Llama 2 (7B) model
42% accuracy on a tiny, outdated model - surely it would improve significantly by fine-tuning Llama 3.1 405B!
Re: Leveraging AI for efficient incident response
#6Im really interested in the implied restriction/focus on “code changes.” IME a very very large number of impacting incidents arent strictly tied to “a” code change, if any at all. It _feels_ like theres an implied solution to tying running version back to deployment rev, to deployment artifacts, and vcs. Boundary conditions and state changes in the distributed system were the biggest bug bear I ran in to at AWS. Then…
Interestingly, with the move to IaC, diagnosing at the level of code change makes increasing sense. It's impressive to see their results given that perspective. Not obvious!
Seperately, we have been curious about extending louie.ai to work not just with logs/DBs, but go in the reverse direction ('shift right'): talk directly to a live OSAgent like an EDR or OSQuery, whether on a live system or a cloud image copy. If of interest to any teams, would love to chat.
Re: Leveraging AI for efficient incident response
#7Way back in the day on FB Ads we trained a GBDT on a bunch of features extracted from the diff that had been (post-hoc) identified as the cause of a SEV.
Unlike a modern LLM (or most any non-trivial NN), a GBDT’s feature importance is defensively rigorous.
After floating the results to a few folks up the chain we burned it and forget where.
Re: Leveraging AI for efficient incident response
#8AI 1: This user is suspicious, lock account
User: Ahh, got locked out, contact support and wait
AI 2: The user is not suspicious, unlock account
User: Great, thank you
AI 1: This account is suspicious, lock account
Re: Leveraging AI for efficient incident response
#9This is really cool. My optimistic take on GenAI, at least with regard to software engineering, is that it seems like we're gonna have a lot of the boring / tedious parts of our jobs get a lot easier!
Claude 3.5 Sonnet still can’t cut me a diff summary based on the patch that I’m generally willing to hand in as my own work and it’s by far the best API-mediated, investor-subsidized one.
Forget the diff, I don’t want my name on the natural language summary.
Re: Leveraging AI for efficient incident response
#10Im really interested in the implied restriction/focus on “code changes.” IME a very very large number of impacting incidents arent strictly tied to “a” code change, if any at all. It _feels_ like theres an implied solution to tying running version back to deployment rev, to deployment artifacts, and vcs. Boundary conditions and state changes in the distributed system were the biggest bug bear I ran in to at AWS. Then…
From my experience, the vast majority of reliability issues at Meta come from 3 areas:
- Code changes
- Configuration changes (this includes the equivalent of server topology changes like cloudformation, quota changes)
- Experimentation rollout changes
There has been issues that are external (like user behavior change for new year / world cup final, physical connection between datacenters being severed…) but they tend to be a lot less frequent.
All the 3 big buckets are tied to a single trackable change with an id so this leads to the ability to do those kind of automated root cause analysis at scale.
Now, Meta is mostly a closed loop where all the infra and product is controlled as one entity so those results may not be applicable outside.