This is exactly what we do at OneUptime.com. Show you AI generated possible Incident remediation based on your data + telemetry + code. All of this is 100% open-source.
Leveraging AI for efficient incident response
31–40 of 58 posts
Re: Leveraging AI for efficient incident response
#32I will be more interested to understand how they deal with injection attacks. Any alert where the attacker controls some parts of the text that ends up in the model could be used to either evade it worse use it to hack it. Slack had an issue like that recently.
Re: Leveraging AI for efficient incident response
#33We've open sourced something with similar goals that you can use today: https://github.com/robusta-dev/holmesgpt/ We're taking a slightly different angle than what Facebook published, in that we're primarily using tool calling and observability data to run investigations. What we've released really shines at surfacing up relevant observability data automatically, and we're soon planning to add the change-tracking ele…
Can we see the recording of this webinar somewhere?
Here you go: https://www.youtube.com/live/Jml1hk6I5Wo?si=YbjJKRkO4yf0bOlx
And thanks for submitting!
Re: Leveraging AI for efficient incident response
#34We've shifted our oncall incident response over to mostly AI at this point. And it works quite well. One of the main reasons why this works well is because we feed the models our incident playbooks and response knowledge bases. These playbooks are very carefully written and maintained by people. The current generation of models are pretty much post-human in following them, performing reasoning and suggesting mitigati…
What does AI add to your playooks ?
Re: Leveraging AI for efficient incident response
#35Earlier quoted context omitted.
No I'm not. It's crap. I have about 30 years experience both on hard engineering (electronics) and software engineering particularly on failure analysis and reliability engineering. Most people are lazy and get led astray with false information. This is a very dangerous thing. You need a proper conceptualisation framework like a KT problem analysis to eliminate incorrect causes and keep people thinking rationally and…
Sounds like you're projecting your own laziness and shortcomings on others. This is a tool that seems really helpful considering the alternative is 0%.
Personal insults aside, "seems" requires no evaluation if the success rate is outside what could be considered a sane confidence interval on trust.
I would literally be fired if I implemented this tool.
Re: Leveraging AI for efficient incident response
#36Im really interested in the implied restriction/focus on “code changes.” IME a very very large number of impacting incidents arent strictly tied to “a” code change, if any at all. It _feels_ like theres an implied solution to tying running version back to deployment rev, to deployment artifacts, and vcs. Boundary conditions and state changes in the distributed system were the biggest bug bear I ran in to at AWS. Then…
From my experience, the vast majority of reliability issues at Meta come from 3 areas: - Code changes - Configuration changes (this includes the equivalent of server topology changes like cloudformation, quota changes) - Experimentation rollout changes There has been issues that are external (like user behavior change for new year / world cup final, physical connection between datacenters being severed…) but they ten…
> this leads to the ability to do those kind of automated root cause analysis at scale.
I'm curious how well that works in the situation where your config change or experiment rollout results in a time bomb (e.g. triggered by task restart after software rollout), speaking as someone who just came off an oncall shift where that was one of our more notable outages.
Google also has a ledger of production events which _most_ common infra will write to, but there are so many distinct systems that I would be worried about identifying spurious correlations with completely unrelated products.
> There has been issues that are external (like ... physical connection between datacenters being severed…) but they tend to be a lot less frequent.
That's interesting to hear, because my experience at Google is that we'll see a peering metro being fully isolated from our network at least once a year; smaller fiber cuts that temporarily leave us with a SPOF or with a capacity shortfall happen much much more frequently.
(For a concrete example: a couple months ago, Hurricane Beryl temporarily took a bunch of peering infrastructure in Texas offline.)
Re: Leveraging AI for efficient incident response
#37We've shifted our oncall incident response over to mostly AI at this point. And it works quite well. One of the main reasons why this works well is because we feed the models our incident playbooks and response knowledge bases. These playbooks are very carefully written and maintained by people. The current generation of models are pretty much post-human in following them, performing reasoning and suggesting mitigati…
What does AI add to your playooks ?
I'm guessing the being awake and fresh at 3am within a few seconds of the incident occuring part.
Re: Leveraging AI for efficient incident response
#38I do think AI will automate a lot of the grunt work involved with incidents and make the life of on-call engineers better.
We are currently working on this at: https://github.com/opslane/opslane
We are starting by tackling adding enrichment to your alerts.
Re: Leveraging AI for efficient incident response
#39We've shifted our oncall incident response over to mostly AI at this point. And it works quite well. One of the main reasons why this works well is because we feed the models our incident playbooks and response knowledge bases. These playbooks are very carefully written and maintained by people. The current generation of models are pretty much post-human in following them, performing reasoning and suggesting mitigati…
Expert systems redux? Being able to provide the expertise in form of plain written English (or another language), will at least make it much more feasible to build them up. And it can also meaningfully be consumed by a human. If it works well for incident response, then there are many usecases that are similar - basically most kinds of diagnostics/troubleshooting of systems. At least the relatively bounded ones, wher…
Why won't it hit the same limits of frame problem or qualification problem?
Expert systems failed in part because of the inability to learn, while HVAC is ladder logic, that I honestly haven't spent much time in, LLMs are inductive.
It will be a useful tool, but expert systems had a very restricted solution space.
Re: Leveraging AI for efficient incident response
#40[deleted]