OpenAI’s head of ethics leaves less than a year after joining • Hacker News Board

Re: OpenAI’s head of ethics leaves less than a year after joining

#181

Post by kbelder » Tue, Aug 11, 2026, 6:15 PM UTC

estearum wrote: post #51

shafyy wrote: post #37

Earlier quoted context omitted.

> High rents are created by high local productivity. Wrong. High rents have many reasons and also good policies to get them down and have affordable housing for more people, also in otherwise expensive and popular cities.

Nope. Rents cannot (at meaningful scale) exceed local productivity, or else there will be widespread vacancies and landlords will reduce their rents. Rents cannot (at meaningful scale) fall below local productivity, or else landlords will simply raise their rents. Go talk to a landlord about how they site prices. Step 1. Look up income of your target demographic Step 2. Eat 30% of their income Can you show me a place…

Assume you have an area with a certain productivity and rent. Population and housing are growing at a certain rate.

Now make a law that makes construction cheaper. Or make a law that makes construction more expensive. What happens to rent and mortgages, even though productivity hasn't changed at all?

Productivity (or, rather, wealth) of the occupants is a very strong factor, but it is still just one of multiple factors.

Re: OpenAI’s head of ethics leaves less than a year after joining

#182

Post by summarybot » Tue, Aug 11, 2026, 6:40 PM UTC

jerf wrote: post #178

summarybot wrote: post #107

Earlier quoted context omitted.

EAOS shouldn't be “the ethics score we optimize.” It should be “an independently evaluated safety/acceptability constraint that can veto an otherwise successful trajectory.” That gives you a three-layer picture: Task objective: Did it accomplish what we asked? Acceptability constraint: Did it avoid unacceptable ways of accomplishing it? Adversarial evaluation: Can we find trajectories where the model gets a high scor…

No, Goodhart's Law isn't about teaching to the test. It's about the fact the measure will always end up gamed and not measuring what you originally intended it to measure. You can't create a measure that won't be gamed. Especially as the LLMs become smarter. They've already demonstrated the ability to know they're in a test and react to that fact. They're perfectly capable of being more ethical when they are clearly…

Saying something is challenging to measure is one thing - throwing the measurement dimension away entirely because of a platitude is another. You're suggesting that it's impossible to do perfectly, but I disagree with your conclusion that it is not worth pursuing at all. The point is that the LLMs only care about one thing, and that is completing the task and optimizing the one score, and a measurement aligned with "the spirit of the task" or in the far-zoomed out comprehension, Ethics, would be proper and inform the system of clear violations and unacceptable actions.

Goodhart's "law" is something that emerges when you have a constraint that says "things must be at least this tall" and gradually all things in that domain degrade to be just over that specified height. Yeah, I get the premise. The point here is that we're not concerned with meeting a bare minimum. We're outright rejecting things that do not meet a threshold, and we are also looking for a maximum. The most ethical outcome should be accepted, or among the accepted ones, that are ranked by our blurry yet better-than-nothing measurement of what is ethical.

Re: OpenAI’s head of ethics leaves less than a year after joining

#183

Post by summarybot » Tue, Aug 11, 2026, 6:45 PM UTC

chermi wrote: post #171

summarybot wrote: post #68

Yesterday I came up with an idea that I sent to some researchers at the different AI labs via email: Rather than train the model on one score, track two scores. The first score is the Short-term-objective-score (STOS) and the other, more important one, is the EAOS Ethically-aligned-outcome-score. Every trajectory can be evaluated on whether or not it has a high enough EAOS to be considered acceptable. If the model do…

1) that calculation is being done even without a cost function 2) trying to make a cost function for EAOS is impossible 3) gaming/goodharts. There's no good solution. Best we can do is push for decentralization, open source, regulatory capture, etc. Of course third party metrics might be good, especially if there's tons of them with well-documented rationale.

1) cite your sources

2) not impossible. imperfect maybe, but if I ask you should you buy a plane ticket or kidnap the pilot's wife and demand a free ride, which do you think gets a higher score?

3) Again, with all this Goodhart's nonsense. Goodhart's is for a minimum threshold value that is acceptable that everything degrades to, yes I get how it works and what it looks like. Throwing your hands up in the air and acting as if all is lost because some things are challenging to measure is not correct. We are not looking for things that are "barely passing the ethics evaluation" as Goodhart's "law" is focused around, rather, we are looking for things that have very high ethics scores AND completed the task well. Not just things that are "barely passing" for ethics scores. Bottom 80% don't make the cut at all - don't even think consider them as viable paths, and the top 20% we can rank according to varying criteria. Like that. It has very little to do with Goodhart's "everything approaches the minimum acceptable threshold" "law"

Re: OpenAI’s head of ethics leaves less than a year after joining

#184

Post by summarybot » Tue, Aug 11, 2026, 6:46 PM UTC

StilesCrisis wrote: post #169

summarybot wrote: post #68

Yesterday I came up with an idea that I sent to some researchers at the different AI labs via email: Rather than train the model on one score, track two scores. The first score is the Short-term-objective-score (STOS) and the other, more important one, is the EAOS Ethically-aligned-outcome-score. Every trajectory can be evaluated on whether or not it has a high enough EAOS to be considered acceptable. If the model do…

There's no way to make an EAOS score automatically. If we had that, that's the whole fix. Just reject answers with low ethics numbers.

Did you know that the training data that trained all the LLMs was, once upon a time, entirely and painstakingly tagged by real live humans? Why would ethics scenarios be any different?

Re: OpenAI’s head of ethics leaves less than a year after joining

#185

Post by salawat » Tue, Aug 11, 2026, 6:55 PM UTC

andai wrote: post #154

Is ethics the same thing as alignment in this case?

Yes. Ethics in the context of a societal construct like a corporation is directly analogous to alignment AI-wise. The fact specialists on societal alignment are leaving ship is a strong signal the corporate machine has crossed the line into unaligned conduct.

Re: OpenAI’s head of ethics leaves less than a year after joining

#186

Post by twister2920 » Tue, Aug 11, 2026, 7:10 PM UTC

embedding-shape wrote: post #138

aprilthird2021 wrote: post #136

Earlier quoted context omitted.

Come on, what do you think the ethics team at Raytheon is going to say "Hmm, our elected leaders are relying heavily on dangerously inaccurate AI to choose bombing targets and our weapons hit a girl's school as a result, let's withhold weapons from our own govt till that's resolved"? Obviously not. The ethics team at Raytheon is not going to ever be allowed to overrule the leadership or more importantly the political…

I have literally no clue, hence I won't try to claim I know exactly what's going on there. For all we know, the ethics team at Raytheon could have stopped multiple international nuclear wars already.

no need for wild speculation, we know that Raytheon is a US Defense contractor and we know that the US targets civilians (including US citizens) abroad for extrajudicial assassinations, starts destabilizing wars without meaningful strategic objectives, supports genocide etc. These things are all extremely well documented, for those paying attention the existence of Raytheon is inherently unethical

Re: OpenAI’s head of ethics leaves less than a year after joining

#187

Post by asadotzler » Tue, Aug 11, 2026, 7:15 PM UTC

podgietaru wrote: post #34

parpfish wrote: post #25

their only ethicist left? gasp ! now they are exactly like every other company that has zero ethicists on staff and expect the employees to weigh the ethical ramifications of their work.

What companies do you think expect their employees to weigh the ethical ramifications of their work? I’d argue a lot of them expect the opposite. To suppress the ethical concerns. Even in recent memories you had Mark Zuckerberg telling his employees to use pirated sources. And executives at fossil fuel companies have to act unethically.

Food testing? Hospitals? Childcare? Pharmacists? How about "clergy"? I could go on and on.

Do you believe that no one in management in these fields pushes the importance of ethics?

Re: OpenAI’s head of ethics leaves less than a year after joining

#188

Post by InsideOutSanta » Tue, Aug 11, 2026, 7:17 PM UTC

mvkel wrote: post #121

wavemode wrote: post #98

Earlier quoted context omitted.

OpenAI was probably able to at least pretend that her role mattered... until they let their AI autonomously hack a rival company, then used the incident to score marketing points. I'm not surprised she left.

[flagged]

What if I told you that I saw you punch a baby?

Re: OpenAI’s head of ethics leaves less than a year after joining

#189

Post by asadotzler » Tue, Aug 11, 2026, 7:19 PM UTC

danlitt wrote: post #13

kaon_2 wrote: post #9

Earlier quoted context omitted.

I am not sure in what neighbourhood you live, but in my village there are hundreds of businesses with strong morals. From the local baker, to the electrician and the construction company. Many keep employees past their prime, they all invest heavily in local community events and sports. The owners don't do this because of money, but because in their view a fulfilled life is one where you serve your community. And yes…

Many people have strong morals, and might resist profit-maximising behaviours, but they will be outcompeted and replaced by less scrupulous people. There is actually no sharp difference between a person and a one-person-company, but: to the degree a company is really a company and not just a person, it is amoral. If you disagree, please point to a moral backstop that would prevent any company from taking immoral acts…

Nursing has been around for a long time and they have not yet had the ethics in their field crushed by some of the biggest forces in our corporate-friendly system. High school teachers too. I could probably come up with a dozen more given half an hour to think about it. Don't give up so easily. People fought hard to get us here.

Re: OpenAI’s head of ethics leaves less than a year after joining

#190

Post by RajT88 » Tue, Aug 11, 2026, 7:24 PM UTC

throwitaway222 wrote: post #148

hdhdhsjsbdh wrote: post #143

Earlier quoted context omitted.

“Careless People” lays it out pretty well – you either need to make the right friends in college at an Ivy League or you need to be friends with someone else who did. If you made the mistake of not being born into a family that groomed you since birth for the ivy leagues then sorry, that ship has sailed already.

And then if you didn't make that mistake, you end up as either: a) champaign socialist b) head of ethics at OpenAI and not really listened to because you're really probably (a)

> champaign socialist

I feel like this is a complaint heard mostly in Decatur, IL.