Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SREBench Competition

sreben.ch

Post reply on HN
Print view View original

1–10 of 28 posts

Re: SREBench Competition

#4

Post by jtsaw »

yjftsjthsd-h wrote: post #2

What am I missing? Of course AI is faster than a human; the problem is that I don't trust it to not break things itself.

yeah that makes sense. While the tech develops, our focus is on building an AI agent that can determine the root cause of an issue, which itself is an import step in fixing things.

Re: SREBench Competition

#5

Post by dgl »

Not sure whether there's lots of people trying out commands right now (is it backed by a real k8s cluster?), but some commands are taking over 10 seconds to run. Not really a fair "benchmark" when the system's speed is variable.

I also only got "partially_correct" for some, not sure whether it wanted more detail or just didn't like how I phrased things. Neat though.

                      Success Rate        MTTR (Mean time to Resolution)
  YOU:                50.00 %              1.80 min
  PARITY AI SRE:      70 %                 2 min
At least I'm faster than an AI?

Re: SREBench Competition

#6

Post by jtsaw »

lagichikool wrote: post #3

Should be called k8sbench? Kubernetes is so bad and the questions asked here are such a good example of why. Response: sudo apt-get purge kube*

We're hoping to expand the dataset into other SRE tasks that aren't kubernetes related. It's just what we've focused on for right now.

Re: SREBench Competition

#7

Post by jtsaw »

dgl wrote: post #5

Not sure whether there's lots of people trying out commands right now (is it backed by a real k8s cluster?), but some commands are taking over 10 seconds to run. Not really a fair "benchmark" when the system's speed is variable. I also only got "partially_correct" for some, not sure whether it wanted more detail or just didn't like how I phrased things. Neat though. Success Rate MTTR (Mean time to Resolution) YOU: 50…

hmmm, not sure where the latency is coming from, might need to up the resources.

Re: SREBench Competition

#8

Post by dgl »

jtsaw wrote: post #7
dgl wrote: post #5

Not sure whether there's lots of people trying out commands right now (is it backed by a real k8s cluster?), but some commands are taking over 10 seconds to run. Not really a fair "benchmark" when the system's speed is variable. I also only got "partially_correct" for some, not sure whether it wanted more detail or just didn't like how I phrased things. Neat though. Success Rate MTTR (Mean time to Resolution) YOU: 50…

hmmm, not sure where the latency is coming from, might need to up the resources.

Maybe the AI can work it out?

More seriously usually issues where the observed behaviour is "the system is slow" are harder to root cause than complete outages. It depends partly how good your capacity planning is obviously, but maybe an AI could help with that too.

Re: SREBench Competition

#9

Post by freeplay »

I'm wondering how they're determining a correct answer. I know, for sure, that one of my answers was correct but it was marked incorrect. I'm wondering if I need to include specific keywords in my answer? How detailed do I need to be?

Re: SREBench Competition

#10

Post by jtsaw »

freeplay wrote: post #9

I'm wondering how they're determining a correct answer. I know, for sure, that one of my answers was correct but it was marked incorrect. I'm wondering if I need to include specific keywords in my answer? How detailed do I need to be?

try to be as detailed as possible. With text sometimes it's hard to determine how close your answer was to the correct one.
Post reply on HN