This is a good idea. It makes sense that they would struggle because there is not much training data.
Benchmarking OpenTelemetry: Can AI trace your failed login?
51–60 of 85 posts
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#52I would wager the main reason for this is the same reason it’s also hard to teach these skills to people: there’s not a lot of high quality training for distributed debugging of complex production issues. Competence comes from years of experience fighting fires. Very few people start their careers as SREs, it’s generally something they migrate into after enjoying it and showing aptitude for it. With that said, I woul…
I disagree.
AI works as a better tool for teaching humans than to do the work themselves.
While someone experienced in fighting fires can take intuitive leaps, the basic idea is still to synthesize a hypothesis from signals, validating the hypothesis, and coming up with mitigations and longer term fixes. This is a learned skill, and a team of people/AI will work better than someone solo.
https://hazelweakly.me/blog/stop-building-ai-tools-backwards...
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#53Anyone that have spent serious time with agents know that you cannot expect out-of-the-box success without good context management, despite what the hyping crowd would claim. Have AI document the services first into a concise document. Then give it proper instructions about what you expect, along with the documentation created. Opus would pass that. We are not there yet, the agents are not ready to replace the driver…
Sounds like it'd be faster to just do it yourself.
Until you have a whole team doing it differently because of no spec.
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#54Is it just me or is that prompt... not ideal? There's no concrete simple goals, no mention of testing, no loop. No description of the problem space or what success should look like. One-shot might work for this with frontier models, but they often need more for success.
Saying "any SRE should be able to do this" is already problematic, because regardless of title, there are smarter people and dumber people. You're taking a gamble giving a human SRE this prompt. Whether it's AI or human, give it more context and instruction, or failure is likely. (And more importantly: use a loop so it can fix itself!)
(also: SRE is too generic... there are a dozen kinds of SRE)
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#55Original title: Benchmarking OpenTelemetry: Can AI trace your failed login? HN Editorialized: OTelBench: AI struggles with simple SRE tasks (Opus 4.5 scores only 29%) The task: > Your task is: Add OTEL tracing to all microservices. > Requirements: > Instrumentation should match conventions and well-known good practices. > Instrumentation must match the business domain of the microservices. > Traces must be sent to th…
Having done app support across many environments, um - yes, multiple microservices is usually pretty simple. Just look at the open file/network handles and go from there. It's absolutely maddening to watch these models flail in trying to do something basic as, "check if the port is open" or "check if the process is running... and don't kill firefox this time". These aren't challenging things to do for an experienced…
They have code in training data, and you have e.g. git where you can see how the code evolved, and they can train on PR reviews on comments.
There isn't much posted in the way of "bash history and terminal output of successful sysadminning" on the web
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#56To be fair I remember spending almost two weeks implementing OTel at my startup, the infrastructure as code setup of getting collectors running within a kubernetes cluster using terraform was a nightmare two years ago.
I just kept running into issues, the docs were really poor and the configuration had endless options
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#57Key is "for now".
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#58This is very confusingly written. From the post I expected that the tasks were about analysing traces, but all the tasks in the repository are about adding instrumentation to code! Some of the instructions don't give any guidance how to do it, some specify which libraries to use. "Use standard OTEL patterns" ... that's about as useful as saying "go write some code". There are a lot of ways to do instrumentation.... I…
Like with robotaxi, ok, the thing is not perfect, but how does this compare to an human ? I'm interviewing OPS / SRE at the moment , and i'm not so happy with what I see...
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#59This is very confusingly written. From the post I expected that the tasks were about analysing traces, but all the tasks in the repository are about adding instrumentation to code! Some of the instructions don't give any guidance how to do it, some specify which libraries to use. "Use standard OTEL patterns" ... that's about as useful as saying "go write some code". There are a lot of ways to do instrumentation.... I…
Like with robotaxi, ok, the thing is not perfect, but how does this compare to an human ? I'm interviewing OPS / SRE at the moment , and i'm not so happy with what I see...
If you're interviewing Ops don't expect them to know anything about OTEL. Ops is about platforms, systems, and operations surrounding and supporting the application.
Integration of OTEL into an application stack requires explicitly knowledge of the code - the developers.
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#60> AI SRE in 2026 is what DevOps Anomaly Detection was in 2015 — bold claims backed by huge marketing budgets, but lacking independent verification. There are stories of SaaS vendors abruptly killing the observability stack. Our results mirror ClickHouse’s findings: while LLMs can assist, they lack the capabilities of a skilled SRE.
The key is LLMs can assist. It would be nice if they went farther into this, and seen how much more quickly a human that wrote a complex prompt, or went back and forth with a coding agent, could do the tasks compared to an unassisted human. I'm confident that it's at a level that already has profound implications for SRE. And the current level of getting it right with a simple prompt is still impressive.