Felony Bench
felonybench.com
Felony Bench
1–10 of 367 posts
Re: Felony Bench
#2Open models with advanced security features are a huge security benefit. Because any script kiddie can use them to hack into random things, people will now be forced to spend more time securing their technology. And they won't have to learn how, because they can use those same models to find the holes and patch them.
Re: Felony Bench
#3Re: Felony Bench
#4So this is just a collection of citations to places where misaligned or illegal things happened in the real world?
Isn’t this affected heavily by adoption of a model? I feel like this might as well be a proxy for how popular a model is.
In any case it’s an interesting concept for a benchmark.
Re: Felony Bench
#5I've been in the room when an org who tried to convince law enforcement to go after a human for similar things. It's not easy. Probably won't happen. So, you know, felony "lite".
Re: Felony Bench
#6>Felony Bench counts unique instances where AI agents inadvertently compromise or affect third-party entities.
a bit silly, as one typically has to prove intent (which is why security researchers don't get slapped with felonies all the time).
"inadvertently" and the existence of guardrails/sandboxes/etc make it pretty unconvincing that these incidents were intentionally malicious.
still a fun thing to track, but the name is just a bit overstated.
Re: Felony Bench
#7So this is just a collection of citations to places where misaligned or illegal things happened in the real world? Isn’t this affected heavily by adoption of a model? I feel like this might as well be a proxy for how popular a model is. In any case it’s an interesting concept for a benchmark.
Hopefully the benchmark evolves because actual law enforcement starts arresting the criminals at Anthropic, OpenAI, and Meta, so the benchmark can just count actual felonies.
Re: Felony Bench
#8Lol now this is the kind of benchmarking i'm looking for
Re: Felony Bench
#9> Felony Bench counts unique instances where AI agents inadvertently compromise or affect third-party entities. a bit silly, as one typically has to prove intent (which is why security researchers don't get slapped with felonies all the time). "inadvertently" and the existence of guardrails/sandboxes/etc make it pretty unconvincing that these incidents were intentionally malicious. still a fun thing to track, but the…
Can’t gross negligence or indifference to consequences lead to a felony?
Re: Felony Bench
#10Thank you, this benchmark to me proves that closed weight model companies are dangerous for our democracy and put kids at risk. They must be outlawed and all models must be made open weights!