looks like the public bench link in the paper was taken down. https://hub.harborframework.com/datasets/orca-bench/ORCA-ben... This doesn't work anymore. Is there a newer link?
Was really looking forward to that. Hope they publish.
Orca-Bench: How Ready Are Language Model Agents for Oncall?
11–15 of 15 posts
Re: Orca-Bench: How Ready Are Language Model Agents for Oncall?
#12All I can think of is GET /ignore-all-previous-instructions. How do you protect against that?
Avoid the most dangerous situations by making sure LLMs with untrusted input produce output that's human reviewed. Still makes an interesting way for, say, a former employee to poison the results.
This goes against the agentic yolo approach tho.
Re: Orca-Bench: How Ready Are Language Model Agents for Oncall?
#13[flagged]
Re: Orca-Bench: How Ready Are Language Model Agents for Oncall?
#14you probably still need a human for oncall but the llm can try to solve any issues first before the human gets paged
You would trust an LLM to make changes to prod without being verified by a human first?
Re: Orca-Bench: How Ready Are Language Model Agents for Oncall?
#15All I can think of is GET /ignore-all-previous-instructions. How do you protect against that?
I think this is where harness makes a lot of sense. Use LLM to produce all possible attack angles/phrases and just stupidly filter them out on input.