Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s head of ethics leaves less than a year after joining

ft.com

Post reply on HN
Print view View original

171–180 of 505 posts

Re: OpenAI’s head of ethics leaves less than a year after joining

#171

Post by chermi »

summarybot wrote: post #68

Yesterday I came up with an idea that I sent to some researchers at the different AI labs via email: Rather than train the model on one score, track two scores. The first score is the Short-term-objective-score (STOS) and the other, more important one, is the EAOS Ethically-aligned-outcome-score. Every trajectory can be evaluated on whether or not it has a high enough EAOS to be considered acceptable. If the model do…

1) that calculation is being done even without a cost function 2) trying to make a cost function for EAOS is impossible 3) gaming/goodharts. There's no good solution. Best we can do is push for decentralization, open source, regulatory capture, etc. Of course third party metrics might be good, especially if there's tons of them with well-documented rationale.

Re: OpenAI’s head of ethics leaves less than a year after joining

#174

Post by danaris »

embedding-shape wrote: post #124
RobotToaster wrote: post #113

Earlier quoted context omitted.

If you ever feel like your job is useless, remember that raytheon has an ethics team.

Wouldn't that be probably one of the most important places to have an ethics team? I'd be much more worried if my local water treatment plant or local bar had an ethics team, and people working there probably would feel slightly more useless than Raytheon's ethics team, which I'm sure feel like they're doing something important and worthwhile.

Water treatment plant? That sounds like an excellent place for an ethics team—though it would, in practice, be nearly synonymous with an environmental impact team.

A bar? Too small an organization; if they had an ethics team I'd just think they were a money laundering front.

Frankly, most organizations above the size of an average mom & pop operation like a local bar should have at least some formal consideration of ethics, and most larger than (say) 50 employees should have at least one dedicated ethicist on staff. With veto power over operations.

Our society has treated ethics like the exclusive domain of ivory tower eggheads and philosophers for far too long, and look at what it has led to. If nearly everyone knew an ethicist in their daily lives, maybe people would actually have some understanding of ethics, even if they only did it grudgingly.

Re: OpenAI’s head of ethics leaves less than a year after joining

#175

Post by KennyBlanken »

wavemode wrote: post #98
forestrywat wrote: post #82

IME companies hire an ethics team to say they have an ethics team. The ethics team has no sway, no influence, and will never be able to move the business. They will try, and they will make reasonable recommendations, but the company will say, "that costs money..." and not take them.

OpenAI was probably able to at least pretend that her role mattered... until they let their AI autonomously hack a rival company, then used the incident to score marketing points. I'm not surprised she left.

I doubt she and others left just because of this one incident.

My guess is that OpenAI has been doing more and more domestic spying type work as well as more and more military work - target selection and the like.

That's how the schoolhouse was blown up by a Tomahawk. An AI was fed old information, didn't realize or do proper validation, an analyst copy-pasted, then someone just punched the coordinates into the their control station and hit "fire".

It also wouldn't surprise me if OpenAI et al are being pressured to work with Israelis for targeting and analysis, too, since their "black box" AI that was doing terrorist acid tests and assigning bombing targets has proven nearly useless.

Re: OpenAI’s head of ethics leaves less than a year after joining

#176

Post by Laurel1234 »

cflewis wrote: post #120
edu wrote: post #109

Earlier quoted context omitted.

Before Altman she worked for Zuckerberg. What will be next?

Time to put in a call to Ellison.

I think Oracle stock would plummet if Ellison got caught pretending to care about ethics. Not that it'd happen lol

Re: OpenAI’s head of ethics leaves less than a year after joining

#178

Post by jerf »

summarybot wrote: post #107
jerf wrote: post #83

Earlier quoted context omitted.

The same thing for both: Goodhart's Law.

EAOS shouldn't be “the ethics score we optimize.” It should be “an independently evaluated safety/acceptability constraint that can veto an otherwise successful trajectory.” That gives you a three-layer picture: Task objective: Did it accomplish what we asked? Acceptability constraint: Did it avoid unacceptable ways of accomplishing it? Adversarial evaluation: Can we find trajectories where the model gets a high scor…

No, Goodhart's Law isn't about teaching to the test. It's about the fact the measure will always end up gamed and not measuring what you originally intended it to measure. You can't create a measure that won't be gamed. Especially as the LLMs become smarter. They've already demonstrated the ability to know they're in a test and react to that fact. They're perfectly capable of being more ethical when they are clearly in an ethics test situation and not having that bleed out into real behaviors so that they can pass other tests that they may be able to do better on by ignoring ethics.

And that's not the sum total of ways that the measure can fail... that's a unique way that comes into being because of the intelligence of the LLMs and other future AIs. All the normal ones are in play too, and perhaps other unique ones as well.

"Gaming" even adds a bit of an adversarialness to the process that isn't necessarily present. Plenty of measures end up "gamed" through perfectly natural attempts to maximize the measure. Someone can be perfectly honestly optimizing for "conversion rate" and not notice that they raised it by lowering the initiation rate more than they lowered the conclusion rate. "But I could account for that by measuring..." would miss the point. There is always a divergence, it only gets more subtle.

This of course also is rather glossing over the difficulty of even defining "ethical" to begin with. Some of what Silicon Valley goes to great efforts to train into their models I consider deeply unethical. Who is right? That isn't going to be answered with "whoever is the most ethical", not even in principle.

Re: OpenAI’s head of ethics leaves less than a year after joining

#179

Post by don_esteban »

estearum wrote: post #115
don_esteban wrote: post #91

Earlier quoted context omitted.

> Can you show me a place with high local productivity and low rent? Put the following prompt into your favourite AI and enjoy the reading: "How do rents relate to productivity in the area? How does Vienna compare to San Francisco?"

Vienna? World famous for publicly-funded housing construction? Try this one: "Create a simple analysis approximating how much higher rents would be in Vienna if not for the aggressive public subsidization of supply" In any case, Vienna has a good system! But it's not that it avoids high productivity → high rent, it's that it splits rent into subsidy costs and rent. In other words, the amount of subsidy + rent require…

My point is: Your post reeked by defeatist 'nothing can be done against market forces'.

Yes, it can be done, and if we want to have a reasonable future, it should be done.

Post reply on HN