Live data from Hacker News

Leveraging AI for efficient incident response

engineering.fb.com

51–58 of 58 posts

Re: Leveraging AI for efficient incident response

#51

Earlier quoted context omitted.

What does AI add to your playooks ?

I'm guessing the being awake and fresh at 3am within a few seconds of the incident occuring part.

I can execute a playbook without AI at 3am in a few seconds using some orchestration tools. Without any AI.

Re: Leveraging AI for efficient incident response

#52

Earlier quoted context omitted.

I'm guessing the being awake and fresh at 3am within a few seconds of the incident occuring part.

I can execute a playbook without AI at 3am in a few seconds using some orchestration tools. Without any AI.

Are you happy about waking up to do so?

Re: Leveraging AI for efficient incident response

#53
post #20
post #4

Im really interested in the implied restriction/focus on “code changes.” IME a very very large number of impacting incidents arent strictly tied to “a” code change, if any at all. It _feels_ like theres an implied solution to tying running version back to deployment rev, to deployment artifacts, and vcs. Boundary conditions and state changes in the distributed system were the biggest bug bear I ran in to at AWS. Then…

> IME a very very large number of impacting incidents arent strictly tied to “a” code change, if any at all Usually this implies there are bigger problems. If something keeps breaking without any change (config / code) then it was likely always broken and just ignored. So when companies do have most of the low hanging fruit resolved it's the changes that break things. I've seen places where everything is duck taped t…

See other child reply upthread, lots of service-to-service style interactions that look more like distributed state than a CR. And my view was across an org scope where even “infrequent” quickly accumulated. AWS is on the order of 50,000 SDEs, running 300 public services (plus a multiple more internal), and each team/microservice with 50 independent deployment targets.

Re: Leveraging AI for efficient incident response

#54

Earlier quoted context omitted.

I can execute a playbook without AI at 3am in a few seconds using some orchestration tools. Without any AI.

Are you happy about waking up to do so?

If you get compensation for being on-call then why not? Unless it’s on Holiday eve

Re: Leveraging AI for efficient incident response

#55

Earlier quoted context omitted.

I can execute a playbook without AI at 3am in a few seconds using some orchestration tools. Without any AI.

Are you happy about waking up to do so?

Been automatically executing playbooks (Ansible) since before you were born. I sleep fine.

This is standard SRe/ Ops practice. Monitoring system detects failures and automatically runs remediation.

You didn’t read the part where I said “using orchestration to tools”.

Re: Leveraging AI for efficient incident response

#56
post #5

> The biggest lever to achieving 42% accuracy was fine-tuning a Llama 2 (7B) model 42% accuracy on a tiny, outdated model - surely it would improve significantly by fine-tuning Llama 3.1 405B!

Yes very interesting potential, it looks like it can be increased in accuracy considerably because Llama 3.1 with 405B parameters has very similar performance with the latest GPT-4o.

Re: Leveraging AI for efficient incident response

#57

Earlier quoted context omitted.

Are you happy about waking up to do so?

Been automatically executing playbooks (Ansible) since before you were born. I sleep fine. This is standard SRe/ Ops practice. Monitoring system detects failures and automatically runs remediation. You didn’t read the part where I said “using orchestration to tools”.

> Been automatically executing playbooks (Ansible) since before you were born.

This made me look up how old Ansible was.

> Initial release: February 20, 2012; 12 years ago

https://en.m.wikipedia.org/wiki/Ansible_(software)

Post reply on HN