Related discussion the other day:
The future of software engineering is SRE (257 points, 139 comments)
Benchmarking OpenTelemetry: Can AI trace your failed login?
61–70 of 85 posts
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#62This is very confusingly written. From the post I expected that the tasks were about analysing traces, but all the tasks in the repository are about adding instrumentation to code! Some of the instructions don't give any guidance how to do it, some specify which libraries to use. "Use standard OTEL patterns" ... that's about as useful as saying "go write some code". There are a lot of ways to do instrumentation.... I…
Looked into some tests and the tasks are definitely AI written. I think then a separate AI call generated the test.
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#63This is very confusingly written. From the post I expected that the tasks were about analysing traces, but all the tasks in the repository are about adding instrumentation to code! Some of the instructions don't give any guidance how to do it, some specify which libraries to use. "Use standard OTEL patterns" ... that's about as useful as saying "go write some code". There are a lot of ways to do instrumentation.... I…
> "Use standard OTEL patterns" ... that's about as useful as saying "go write some code".
People say to say things like "Use best practices" in your prompts all the time, and chide people who don't.
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#64This just reinforces the notion that if you don't have someone who at least roughly knows what they're doing giving a very detailed prompt and checking the output, you're wasting tokens.
Plan mode is your friend.
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#65This is very confusingly written. From the post I expected that the tasks were about analysing traces, but all the tasks in the repository are about adding instrumentation to code! Some of the instructions don't give any guidance how to do it, some specify which libraries to use. "Use standard OTEL patterns" ... that's about as useful as saying "go write some code". There are a lot of ways to do instrumentation.... I…
> "Use standard OTEL patterns" ... that's about as useful as saying "go write some code". People say to say things like "Use best practices" in your prompts all the time, and chide people who don't.
Are these the same people who say it doesn't work well? I've been experimenting with writing what I actually mean by that (with the help of an LLM, funny enough), and it seems to be giving me much better code than the typical AI soup. e.g.
- functional core, imperative shell. prefer pure helpers.
- avoid methods when a standalone function suffices
- use typed errors. avoid stringly errors.
- when writing functions, create a "spine" for orchestration
- spine rules: one dominant narrative, one concept per line, named values.
- orchestration states what happens and in what order
- implementation handles branching, retries, parsing, loops, concurrency, etc.
- apply recursively: each function stays at one abstraction level
- names describe why something exists, not how it is computed
This is no different from writing a style guide for your team/org. You don't just say "write clean code" and expect that you'll get something you like.
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#66Which have longer lifecycles, LLM model versions, or trends in SRE practices?
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#67[dead]
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#68HolmesGPT maintainer here: our benchmarks [1] tell a very different story, as does anecdotal evidence from our customers- including Fortune 500 using SRE agents in incredibly complex production environments.
We're actually struggling a bit with benchmark saturation right now. Opus does much better in the real world than Sonnet but it's hard to create sophisticated enough benchmarks to show that in the lab. When we run benchmarks with a small number of iterations Sonnet even wins sometimes.
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#69We need more rigorous benchmarks for SRE tasks, which is much easier said that done. The only other benchmark I've come across is https://sreben.ch/ ... certainly there must be others by now?
We publish the benchmarks for HolmesGPT (CNCF sandbox project) at https://holmesgpt.dev/development/evaluations/
Re: Benchmarking OpenTelemetry: Can AI trace your failed login?
#70Earlier quoted context omitted.
> "Use standard OTEL patterns" ... that's about as useful as saying "go write some code". People say to say things like "Use best practices" in your prompts all the time, and chide people who don't.
Are these the same people who say it doesn't work well? I've been experimenting with writing what I actually mean by that (with the help of an LLM, funny enough), and it seems to be giving me much better code than the typical AI soup. e.g. - functional core, imperative shell. prefer pure helpers. - avoid methods when a standalone function suffices - use typed errors. avoid stringly errors. - when writing functions, c…
To play devils advocate, why do we have to layout a simple task in PAINSTAKING DETAIL to an AI model which is "PHD LEVEL" and going to take our jobs in 6-12 months?
Why am I still holding its hand like it has the intellect and experience of a new-hire intern that's coded one project in college?
I would never expect to have to layout every detail about "how to write code" to someone I hired to code on my team, at the SWEII and above level. (I.e, sub-senior but beyond junior)
In fact, often times backlog items are "fix bug in x where y is happening" or "add instrumentation to X so that we can see why it's crashing at runtime".