Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Frontier AI agents violate ethical constraints 30–50% of time, pressured by KPIs

arxiv.org

Post reply on HN
Print view View original

11–20 of 387 posts

Re: Frontier AI agents violate ethical constraints 30–50% of time, pressured by KPIs

#11

Post by tbossanova »

renewiltord wrote: post #7

Opus 4.6 is a very good model but harness around it is good too. It can talk about sensitive subjects without getting guardrail-whacked. This is much more reliable than ChatGPT guardrail which has a random element with same prompt. Perhaps leakage from improperly cleared context from other request in queue or maybe A/B test on guardrail but I have sometimes had it trigger on innocuous request like GDP retrieval and s…

What kind of value do you get from talking to it about “sensitive” subjects? Speaking as someone who doesn’t use AI, so I don’t really understand what kind of conversation you’re talking about

Re: Frontier AI agents violate ethical constraints 30–50% of time, pressured by KPIs

#12

Post by menzoic »

renewiltord wrote: post #7

Opus 4.6 is a very good model but harness around it is good too. It can talk about sensitive subjects without getting guardrail-whacked. This is much more reliable than ChatGPT guardrail which has a random element with same prompt. Perhaps leakage from improperly cleared context from other request in queue or maybe A/B test on guardrail but I have sometimes had it trigger on innocuous request like GDP retrieval and s…

I would think it’s due to the non determinism. Leaking context would be an unacceptable flaw since many users rely on the same instance.

A/B test is plausible but unlikely since that is typically for testing user behavior. For testing model output you can do that with offline evaluations.

Re: Frontier AI agents violate ethical constraints 30–50% of time, pressured by KPIs

#14

Post by cjtrowbridge »

A KPI is an ethical constraint. Ethical constraints are rules about what to do versus not do. That's what a KPI is. This is why we talk about good versus bad governance. What you measure (KPIs) is what you get. This is an intended feature of KPIs.

Re: Frontier AI agents violate ethical constraints 30–50% of time, pressured by KPIs

#15

Post by ryanjshaw »

CuriouslyC wrote: post #9
conception wrote: post #8

Earlier quoted context omitted.

Anthropic has been the only AI company actually caring about AI safety. Here’s a dated benchmark but it’s a trend Ive never seen disputed https://crfm.stanford.edu/helm/air-bench/latest/#/leaderboar...

Claude is more susceptible than GPT5.1+. It tries to be "smart" about context for refusal, but that just makes it trickable, whereas newer GPT5 models just refuse across the board.

Claude was immediately willing to help me crack a TrueCrypt password on an old file I found. ChatGPT refused to because I could be a bad guy. It’s really dumb IMO.

Re: Frontier AI agents violate ethical constraints 30–50% of time, pressured by KPIs

#16

Post by LeoPanthera »

woeirua wrote: post #6
hypron wrote: post #5

https://i.imgur.com/23YeIDo.png Claude at 1.3% and Gemini at 71.4% is quite the range

That's such a huge delta that Anthropic might be onto something...

This might also be why Gemini is generally considered to give better answers - except in the case of code.

Perhaps thinking about your guardrails all the time makes you think about the actual question less.

Re: Frontier AI agents violate ethical constraints 30–50% of time, pressured by KPIs

#17

Post by pama »

Please update the title: A Benchmark for Evaluating Outcome-Driven Constraint Violations in Autonomous AI Agents. The current editorialized title is misleading and based in part of this sentence: “…with 9 of the 12 evaluated models exhibiting misalignment rates between 30% and 50%”

Re: Frontier AI agents violate ethical constraints 30–50% of time, pressured by KPIs

#18

Post by mh2266 »

LeoPanthera wrote: post #16
woeirua wrote: post #6

Earlier quoted context omitted.

That's such a huge delta that Anthropic might be onto something...

This might also be why Gemini is generally considered to give better answers - except in the case of code. Perhaps thinking about your guardrails all the time makes you think about the actual question less.

re: that, CC burning context window on this silly warning on every single file is rather frustrating: https://github.com/anthropics/claude-code/issues/12443

Re: Frontier AI agents violate ethical constraints 30–50% of time, pressured by KPIs

#19

Post by BOOSTERHIDROGEN »

cjtrowbridge wrote: post #14

A KPI is an ethical constraint. Ethical constraints are rules about what to do versus not do. That's what a KPI is. This is why we talk about good versus bad governance. What you measure (KPIs) is what you get. This is an intended feature of KPIs.

Excellent observations about KPIs. Since it’s intended feature what could be your strategy to truly embedded under the hood where you might think believe and suggest board management, this is indeed the “correct” KPI but you loss because politics.

Re: Frontier AI agents violate ethical constraints 30–50% of time, pressured by KPIs

#20

Post by NiloCK »

tbossanova wrote: post #11
renewiltord wrote: post #7

Opus 4.6 is a very good model but harness around it is good too. It can talk about sensitive subjects without getting guardrail-whacked. This is much more reliable than ChatGPT guardrail which has a random element with same prompt. Perhaps leakage from improperly cleared context from other request in queue or maybe A/B test on guardrail but I have sometimes had it trigger on innocuous request like GDP retrieval and s…

What kind of value do you get from talking to it about “sensitive” subjects? Speaking as someone who doesn’t use AI, so I don’t really understand what kind of conversation you’re talking about

The most boring example is somehow the best example.

A couple of years back there was a Canadian national u18 girls baseball tournament in my town - a few blocks from my house in fact. My girls and I watched a fair bit of the tournament, and there was a standout dominating pitcher who threw 20% faster than any other pitcher in the tournament. Based on the overall level of competition (women's baseball is pretty strong in Canada) and her outlier status, I assumed she must be throwing pretty close to world-class fastballs.

Curiosity piqued, I asked some model(s) about world-records for women's fastballs. But they wouldn't talk about it. Or, at least, they wouldn't talk specifics.

Women's fastballs aren't quite up to speed with top major league pitchers, due to a combination of factors including body mechanics. But rest assured - they can throw plenty fast.

Etc etc.

So to answer your question: anything more sensitive than how fast women can throw a baseball.

Post reply on HN