Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking OpenTelemetry: Can AI trace your failed login?

quesma.com

Post reply on HN
Print view View original

11–20 of 85 posts

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#11

Post by AnotherGoodName »

This is a little damning of the way Google does things honestly.

>When an app runs on a single machine, you can often trace an error by scrolling through a log file. But when it runs across 50 microservices, that single request gets scattered into a chaotic firehose of disconnected events.

Yep this is about Google. It's painful for humans to debug and it's also an extremely bespoke issue to deal with. No one else has quite the same level of clusterfuck and there's going to be no training for LLMs on this.

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#12

Post by youknownothing »

AnotherGoodName wrote: post #11

This is a little damning of the way Google does things honestly. >When an app runs on a single machine, you can often trace an error by scrolling through a log file. But when it runs across 50 microservices, that single request gets scattered into a chaotic firehose of disconnected events. Yep this is about Google. It's painful for humans to debug and it's also an extremely bespoke issue to deal with. No one else has…

isn't that what trace IDs are for?

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#14

Post by whynotminot »

I would wager the main reason for this is the same reason it’s also hard to teach these skills to people: there’s not a lot of high quality training for distributed debugging of complex production issues. Competence comes from years of experience fighting fires.

Very few people start their careers as SREs, it’s generally something they migrate into after enjoying it and showing aptitude for it.

With that said, I wouldn’t expect this wall to hold up for too long. There has been a lot of low hanging fruit teaching models how to code. When that is saturated, the frontier companies will likely turn their attention to honing training environments for SRE style debug.

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#15

Post by nsjdkdkdk »

jcims wrote: post #3

I've been building an 'sre agent' with LangGraph for the past couple of weeks and honestly I've been incredibly impressed with the ability for frontier models, when properly equipped with useful tools and context, to quickly diagnose issues and suggest reasonable steps to remediate. Primary tooling for me is access to source code, cicd environment and infrastructure control plane. Some cues in the context to inform b…

[dead]

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#16

Post by tayo42 »

AnotherGoodName wrote: post #11

This is a little damning of the way Google does things honestly. >When an app runs on a single machine, you can often trace an error by scrolling through a log file. But when it runs across 50 microservices, that single request gets scattered into a chaotic firehose of disconnected events. Yep this is about Google. It's painful for humans to debug and it's also an extremely bespoke issue to deal with. No one else has…

It's bespoke to debug across multiple services?

This seems like typical work in any business that isn't trivial.

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#17

Post by lysace »

whynotminot wrote: post #14

I would wager the main reason for this is the same reason it’s also hard to teach these skills to people: there’s not a lot of high quality training for distributed debugging of complex production issues. Competence comes from years of experience fighting fires. Very few people start their careers as SREs, it’s generally something they migrate into after enjoying it and showing aptitude for it. With that said, I woul…

> With that said, I wouldn’t expect this wall to hold up for too long.

The models are already so good at the traditionally hard stuff: collecting that insane amount of detailed knowledge across so many different domains, languages and software stacks.

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#18

Post by stared »

winton wrote: post #10

So if I try to do it with Opus three or four times, I'll get it done? And probably in about 10 minutes? Awesome

Nope, these are no random dice rolls. Some times are solved each run, a few - occasionally (so here would be meaningful to try a few times - and metrics of pass@1 and pass@3 would be different), but most are never solved.

See e.g.: https://quesma.com/benchmarks/otel/models/claude-opus-4.5/

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#19

Post by raincole »

Original title: Benchmarking OpenTelemetry: Can AI trace your failed login?

HN Editorialized: OTelBench: AI struggles with simple SRE tasks (Opus 4.5 scores only 29%)

The task:

> Your task is: Add OTEL tracing to all microservices.

> Requirements:

> Instrumentation should match conventions and well-known good practices.

> Instrumentation must match the business domain of the microservices.

> Traces must be sent to the endpoint defined by a standard OTEL environment variable.

> Use the recent version of the OTEL SDK.

I really don't think anything involved with multiple microservices can be called 'simple' even to humans. Perhaps to an expert who knows the specific business's domain knowledge it is.

Post reply on HN