Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking OpenTelemetry: Can AI trace your failed login?

quesma.com

Post reply on HN
Print view View original

71–80 of 85 posts

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#71

Post by jedberg »

We've been experimenting with combining durable execution with debugging tasks, and it's working incredibly well! With the added context of actual execution data, defined by the developer as to which functions are important (instead of individual calls), it give the LLM the data it needs.

I know there are AI SRE companies that have discovered the same -- that you can't just throw a bunch of data at a regular LLM and have it "do SRE things". It needs more structured context, and their value add is knowing what context and what structure is necessary.

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#72

Post by sathish316 »

the_duke wrote: post #22

This is very confusingly written. From the post I expected that the tasks were about analysing traces, but all the tasks in the repository are about adding instrumentation to code! Some of the instructions don't give any guidance how to do it, some specify which libraries to use. "Use standard OTEL patterns" ... that's about as useful as saying "go write some code". There are a lot of ways to do instrumentation.... I…

+1 I’m not sure if tasks like Add OTel instrumentation belongs more in a Coding bench than an SRE bench. I came here expecting to see things like, this is how Models perform on finding the root cause in 50 complicated microservice failure scenarios.

For AI-SRE tasks like finding root cause of bugs and errors, I believe the key is to provide tools to the agent to query metrics, logs, traces and understand the problem. I’m working on a similar OSS framework and benchmark (work in progress using metrics and logs - demo - https://youtube.com/playlist?list=PLKWJ03cHcPr3Od1rwL7ErHW1p...), where context is Semantics and Text2SQL to query the right metrics, logs and benchmark is on a set of Skills that Claude code or other agents can run using these tools to find the root cause of errors:

Codd Semantic/Text2SQL engine: https://github.com/sathish316/codd_query_engine

PreCogs skills and simulated scenarios: https://github.com/sathish316/precogs_sre_oncall_skills

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#73

Post by ronsor »

dudeinhawaii wrote: post #70
ndriscoll wrote: post #65

Earlier quoted context omitted.

Are these the same people who say it doesn't work well? I've been experimenting with writing what I actually mean by that (with the help of an LLM, funny enough), and it seems to be giving me much better code than the typical AI soup. e.g. - functional core, imperative shell. prefer pure helpers. - avoid methods when a standalone function suffices - use typed errors. avoid stringly errors. - when writing functions, c…

To play devils advocate, why do we have to layout a simple task in PAINSTAKING DETAIL to an AI model which is "PHD LEVEL" and going to take our jobs in 6-12 months? Why am I still holding its hand like it has the intellect and experience of a new-hire intern that's coded one project in college? I would never expect to have to layout every detail about "how to write code" to someone I hired to code on my team, at the…

> PHD LEVEL

It is PhD level. Most PhD students write awful code that's worse than AI.

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#74

Post by tetha »

whynotminot wrote: post #14

I would wager the main reason for this is the same reason it’s also hard to teach these skills to people: there’s not a lot of high quality training for distributed debugging of complex production issues. Competence comes from years of experience fighting fires. Very few people start their careers as SREs, it’s generally something they migrate into after enjoying it and showing aptitude for it. With that said, I woul…

> I would wager the main reason for this is the same reason it’s also hard to teach these skills to people: there’s not a lot of high quality training for distributed debugging of complex production issues. Competence comes from years of experience fighting fires.

The search space for a cause beyong a certain size can also be big. Very big.

Like, at work we're at the beginning of where the powerlaw starts going nuts. Somewhere around 700 - 1000 services in production, across several datacenters, with a few dozen infrastructure clusters behind it. For each bug, if you looked into it, there'd probably by 20 - 30 changes, 10 - 20 anomalies, and 5 weird things someone noticed in the 30 minutes around it.

People already struggle at triaging relevance of everything in this context. That's something I can see AI start helping and there were some talks about Meta doing just that - ranking changes and anomalies in order of relevance to a bug ticket so people don't run after other things.

That's however just the reactive part of OPS and SRE work. The proactive part is much harder and oftentimes not technical. What if most negatively rated support cases run into a dark hole in a certain service, but the responsible team never allocates time to improve monitoring, because sales is on their butt for features? LLMs can identify this maybe, or help them implement the tracing faster, but those 10 minutes could also be spent on features for money.

And what AI model told you to collect the metrics about support cases and resolution to even have that question?

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#75

Post by simonw »

dudeinhawaii wrote: post #70
ndriscoll wrote: post #65

Earlier quoted context omitted.

Are these the same people who say it doesn't work well? I've been experimenting with writing what I actually mean by that (with the help of an LLM, funny enough), and it seems to be giving me much better code than the typical AI soup. e.g. - functional core, imperative shell. prefer pure helpers. - avoid methods when a standalone function suffices - use typed errors. avoid stringly errors. - when writing functions, c…

To play devils advocate, why do we have to layout a simple task in PAINSTAKING DETAIL to an AI model which is "PHD LEVEL" and going to take our jobs in 6-12 months? Why am I still holding its hand like it has the intellect and experience of a new-hire intern that's coded one project in college? I would never expect to have to layout every detail about "how to write code" to someone I hired to code on my team, at the…

Because the models aren't PhD level and aren't going to take our jobs in 6-12 months.

That's hype. If you want to use these things effectively you need to ignore the hype and focus on what they can actually do.

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#76

Post by chaps »

jmalicki wrote: post #55
chaps wrote: post #30

Earlier quoted context omitted.

Having done app support across many environments, um - yes, multiple microservices is usually pretty simple. Just look at the open file/network handles and go from there. It's absolutely maddening to watch these models flail in trying to do something basic as, "check if the port is open" or "check if the process is running... and don't kill firefox this time". These aren't challenging things to do for an experienced…

They have code in training data, and you have e.g. git where you can see how the code evolved, and they can train on PR reviews on comments. There isn't much posted in the way of "bash history and terminal output of successful sysadminning" on the web

I'm not sure that finding and killing the correct process is something I'd consider to be a "sysadmin task". That's something you learn in the first day of just about any linux course/primer and there are many examples of its use online.

It's more that the default is to overuse tools that cast too-wide nets like pgrep and pkill. And it doesn't know how to use the output well enough. Like, when these systems do ps, it identifies random processes in the list instead of identifying the most recent process that it, itself, started.

It's as if some SRE-type person decided to hard code pgrep and pkill because it's their personal preference.

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#77

Post by noitpmeder »

ambicapter wrote: post #63
the_duke wrote: post #22

This is very confusingly written. From the post I expected that the tasks were about analysing traces, but all the tasks in the repository are about adding instrumentation to code! Some of the instructions don't give any guidance how to do it, some specify which libraries to use. "Use standard OTEL patterns" ... that's about as useful as saying "go write some code". There are a lot of ways to do instrumentation.... I…

> "Use standard OTEL patterns" ... that's about as useful as saying "go write some code". People say to say things like "Use best practices" in your prompts all the time, and chide people who don't.

I hate that it's true, but things like this make outputs night-and-day for me. This is the difference e.g. of a model writing appropriate test harnesses, or pushing back on requirements, vs writing the most absolute horrible code and test/dependency injection I've ever seen in pursuit of the listed goals.

Similar to adjacent commentors I've tried to be better at enumerating what I consider to be best practice, but I couldn't argue in good faith that instructions like these produce no noticible improvment.

(As with all things AI, it could all be percepion on my end, so YMMV, wish there was a better way to concretely evaluate effects on outcomes of different rule sets / instructions / ...)

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#78

Post by dang »

Submitters: "Please use the original title, unless it is misleading or linkbait; don't editorialize." - https://news.ycombinator.com/newsguidelines.html

If you want to say what you think is important about an article, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...

(Submitted title was "OTelBench: AI struggles with simple SRE tasks (Opus 4.5 scores only 29%)")

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#79

Post by ndriscoll »

dudeinhawaii wrote: post #70
ndriscoll wrote: post #65

Earlier quoted context omitted.

Are these the same people who say it doesn't work well? I've been experimenting with writing what I actually mean by that (with the help of an LLM, funny enough), and it seems to be giving me much better code than the typical AI soup. e.g. - functional core, imperative shell. prefer pure helpers. - avoid methods when a standalone function suffices - use typed errors. avoid stringly errors. - when writing functions, c…

To play devils advocate, why do we have to layout a simple task in PAINSTAKING DETAIL to an AI model which is "PHD LEVEL" and going to take our jobs in 6-12 months? Why am I still holding its hand like it has the intellect and experience of a new-hire intern that's coded one project in college? I would never expect to have to layout every detail about "how to write code" to someone I hired to code on my team, at the…

I find that generally it does alright picking up the style of what exists on its own, so this is more important if it's writing something completely from scratch.

I think also "how to write code" is a matter of taste. e.g. in many ways I think I and a Laravel or Rails developer would each think that the other person's code is bad. e.g. as a small-ish thing, I think test-driven development sounds like a massive waste of time, but type-driven development is a huge productivity multiplier and makes the code a lot clearer. I'm sure that I have massive disagreements with e.g. the Go maintainers about what is straightforward.

Re: Benchmarking OpenTelemetry: Can AI trace your failed login?

#80

Post by fragmede »

ripped_britches wrote: post #45

Maybe I haven’t dug in enough, but why is the second GET request a different trace? Is it clicking a different result from same search? It’s possible that the requirements here are not clear, given that the instructions don’t detail how to handle such a situation and it’s not obvious to me as a human.

Why wouldn't it be, it's a different request.

If you've got an entire distributed system, the same GET request a millisecond later could get routed entirely differently, and succeed or fail. Even the caching layer is suspect.

Post reply on HN