Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SREBench Competition

sreben.ch

Post reply on HN
Print view View original

21–28 of 28 posts

Re: SREBench Competition

#21

Post by jtsaw »

deathanatos wrote: post #20

~So like, what am I missing?~ (edit: I'm not missing anything; an AI still can't do my job.) Pod is stuck in 'ContainerCreating' state and never starts. $ kubectl get po -A ``` NAMESPACE NAME READY STATUS RESTARTS AGE default my-app-5d8d6f6d6f-abcde 1/1 Running 0 2d default my-app-5d8d6f6d6f-fghij 1/1 Running 0 2d kube-system coredns-558bd4d5db-xyz12 1/1 Running 0 5d kube-system coredns-558bd4d5db-xyz34 1/1 Running 0…

yea, we'd like to actually create these issues on a real cluster, but we couldn't figure out a good way of doing it at scale. The best alternative that we could think of was using an LLM that knows the root cause and could hopefully simulate outputs of commands consistently. Let us know if you have other ideas, we're always looking for ways to improve it.

Re: SREBench Competition

#24

Post by guessmyname »

For the user registration (if you want to get the Amazon gift card) you send a request like this:

  POST /api/trpc/submitUser?batch=1 HTTP/2.0
  Host: sreben.ch
  Cookie: 
  Trpc-Accept: application/jsonl
  Content-Type: application/json
  Referer: https://sreben.ch/race
  
  {"0":{"name":"","email":"","role":"","company":""}}
Then, requests like this will grade your answers:

  POST /api/trpc/gradeOutput?batch=1 HTTP/2
  Host: sreben.ch
  Cookie: 
  Trpc-Accept: application/jsonl
  Content-Type: application/json
  Referer: https://sreben.ch/race
  
  {"0":{"userRootCause":"","testNumber":}}
Specially useful if the “Submit Root Cause” button doesn’t work for you either.

Also, make sure to type the entire error message, e.g. “ERROR Application performance degraded due to CPU throttling” instead of simply “CPU throttling”, otherwise, you’ll get a "partially_correct" grade.

Re: SREBench Competition

#26

Post by mjlee »

jtsaw wrote: post #21
deathanatos wrote: post #20

~So like, what am I missing?~ (edit: I'm not missing anything; an AI still can't do my job.) Pod is stuck in 'ContainerCreating' state and never starts. $ kubectl get po -A ``` NAMESPACE NAME READY STATUS RESTARTS AGE default my-app-5d8d6f6d6f-abcde 1/1 Running 0 2d default my-app-5d8d6f6d6f-fghij 1/1 Running 0 2d kube-system coredns-558bd4d5db-xyz12 1/1 Running 0 5d kube-system coredns-558bd4d5db-xyz34 1/1 Running 0…

yea, we'd like to actually create these issues on a real cluster, but we couldn't figure out a good way of doing it at scale. The best alternative that we could think of was using an LLM that knows the root cause and could hopefully simulate outputs of commands consistently. Let us know if you have other ideas, we're always looking for ways to improve it.

Would Kubernetes in Docker help? https://kind.sigs.k8s.io/

Re: SREBench Competition

#28

Post by thinkmassive »

jtsaw wrote: post #7
dgl wrote: post #5

Not sure whether there's lots of people trying out commands right now (is it backed by a real k8s cluster?), but some commands are taking over 10 seconds to run. Not really a fair "benchmark" when the system's speed is variable. I also only got "partially_correct" for some, not sure whether it wanted more detail or just didn't like how I phrased things. Neat though. Success Rate MTTR (Mean time to Resolution) YOU: 50…

hmmm, not sure where the latency is coming from, might need to up the resources.

If only there were a software engineering discipline that focused on these types of issues /s
Post reply on HN