Faulty Reward Functions in the Wild
openai.com
Faulty Reward Functions in the Wild
1–10 of 19 posts
Re: Faulty Reward Functions in the Wild
#2This reminds me of how metric-driven companies can go off the rails when they over-optimized for metrics that almost, but not perfectly describe their actual goals.
Re: Faulty Reward Functions in the Wild
#3This reminds me of how metric-driven companies can go off the rails when they over-optimized for metrics that almost, but not perfectly describe their actual goals.
> metric-driven companies can go off the rails
Any publicly traded corporation (save a small handful with a non-traditional governance model) are metric-driven companies.
Modern corporations are paperclip maximizer functions executing on a network general-purpose biological computational engines tied together with powerpoint and email and excel spreadsheets.
Want to know what the AI of the future will look like? It will be a lot like Comcast, because it will be built by Comcast and harnessed to the corporate goals of Comcast and thus will have the same value system as Comcast.
The only thing it will lack is Comcast's institutional incompetence, as it will be Comcast's goals executing on dedicated hardware and not semi-autonomous employees. And it will build a dedicated model of every man and woman on the planet, and use that information to build a personalized profile that will determine exactly how many illegitimate charges it can cram on your bill before you'll suffer through a customized cancellation service that is calibrated to your personality and mental state to be just painful enough to drive you to the brink of suicide. And the only reason it's merely to be brink, is because a dead customer is an unprofitable one. (and if you think that's hyperbole, you have a far brighter view of the future than I do)
Re: Faulty Reward Functions in the Wild
#4This reminds me of how metric-driven companies can go off the rails when they over-optimized for metrics that almost, but not perfectly describe their actual goals.
> metric-driven companies can go off the rails Any publicly traded corporation (save a small handful with a non-traditional governance model) are metric-driven companies. Modern corporations are paperclip maximizer functions executing on a network general-purpose biological computational engines tied together with powerpoint and email and excel spreadsheets. Want to know what the AI of the future will look like? It w…
And I thought I was pessimistic. I must be missing something, how did we go from a faulty code, to KPIs, to Comcast paving the future of the AI?
Re: Faulty Reward Functions in the Wild
#5Earlier quoted context omitted.
> metric-driven companies can go off the rails Any publicly traded corporation (save a small handful with a non-traditional governance model) are metric-driven companies. Modern corporations are paperclip maximizer functions executing on a network general-purpose biological computational engines tied together with powerpoint and email and excel spreadsheets. Want to know what the AI of the future will look like? It w…
And I thought I was pessimistic. I must be missing something, how did we go from a faulty code, to KPIs, to Comcast paving the future of the AI?
> how did we go from a faulty code
The code in the example isn't faulty. The goals are faulty in an non-obvious way, and drives the AI to optimize a solution to a fitness function in a way that humans might consider pathological behavior.
> to Comcast paving the future of the AI?
Comcast's goals are faulty in a non-obvious way. Comcast's goals are to maximize shareholder profit, and in doing so creates a culture of fraudulent billing and a horrifying gauntlet of a cancellation process that humans might consider pathological behavior. And AI is the next logical step to optimizing those corporate values.
Re: Faulty Reward Functions in the Wild
#6This reminds me of how metric-driven companies can go off the rails when they over-optimized for metrics that almost, but not perfectly describe their actual goals.
> metric-driven companies can go off the rails Any publicly traded corporation (save a small handful with a non-traditional governance model) are metric-driven companies. Modern corporations are paperclip maximizer functions executing on a network general-purpose biological computational engines tied together with powerpoint and email and excel spreadsheets. Want to know what the AI of the future will look like? It w…
If I may quote from the rulebook for Fantasy Flight Games's New Angeles board game (go play it, it's awesome!)
There’s something they used to call ‘the Myth of Shareholder Value.’ It goes, basically, like this: everything any corporate officer does, at any level, must be dedicated to only one thing, which is to increase shareholder value.
...
The Weyland Consortium are the high priests of the cult of Shareholder Value in today’s economy. The Consortium is less a corporation and more an algorithm, buying and selling corporations and extending its tendrils out through every sector of industry: research, transportation, you name it. The Consortium moves and acts like a living thing. No one executive or committee steers that ship. Shareholder Value is its only captain, and every decision made by its chief officers is pre-destined, an inevitable result of gears turning for decades, of market forces filtered through AIs running on corporate servers.
Re: Faulty Reward Functions in the Wild
#7This reminds me of how metric-driven companies can go off the rails when they over-optimized for metrics that almost, but not perfectly describe their actual goals.
> metric-driven companies can go off the rails Any publicly traded corporation (save a small handful with a non-traditional governance model) are metric-driven companies. Modern corporations are paperclip maximizer functions executing on a network general-purpose biological computational engines tied together with powerpoint and email and excel spreadsheets. Want to know what the AI of the future will look like? It w…
I don't expect it will go that far at all. Surely a company will test the idea of having an AI give executive-level guidance, but when it does so, it will be hastily dismantled. Companies do not structure themselves in the way they do and act the way they do timidly. The C-level executives are not worrying that they could be doing things better. You can see this clearly as essentially every single large company consciously and intentionally ignores research. There are over a thousand studies showing that open floor plan offices are abyssmal for productivity and actively reduce the profit a company earns. Any AI would tell the execs to have the majority of their workers work remotely, and give the few which remain in company facilities private offices. And the execs would summarily ignore it and cancel the project, declaring it a failure.
They have faith that they are doing the right thing. Research tells them they are not, and they either do not care, or, in my opinion, desperately guard their self image as a 'leader' over and above any concern for profit or long-term viability of the business. An AI would be permitted to take a strong role in making decisions for a company only insofar as it plays the role of toady, having been tweaked and misconfigured to ignore all facts which could result in it telling the executives that they are running 'their company' in the wrong way.
Re: Faulty Reward Functions in the Wild
#8> ...by prioritizing the acquisition of reward signals above
> other measures of success.
This is also true for humans in poorly designed systems. For example, kids become experts at passing tests irrespective of mastering the material. In the workplace, employees become skilled at clocking extra time without finishing additional work. It's reasonable to say that this would eventually emerge in systems which approximate human behavior.
The video shown in the article could just as easily have been a human who just discovered the bug, and wants to troll a bit. The key difference is that a human would soon get bored. Our algorithms don't know about boredom outside of the domain of the reward function.
After playing with a bugged state, A human would lose just enough interest so as to keep playing the game, but without any further interest in the "bugged" state. A human is smart enough to know that there are various microstates of such a "bugged" state, and to ignore those instances as well.
The algorithm is smart enough to find the hack, but it's not smart enough to say "Hey, this is a non-solution, and I am not very happy about that". What is it that makes a human decide to lose interest in such a bugged state? Are these factors locally contained or are they due to external influences?
Re: Faulty Reward Functions in the Wild
#9Earlier quoted context omitted.
> metric-driven companies can go off the rails Any publicly traded corporation (save a small handful with a non-traditional governance model) are metric-driven companies. Modern corporations are paperclip maximizer functions executing on a network general-purpose biological computational engines tied together with powerpoint and email and excel spreadsheets. Want to know what the AI of the future will look like? It w…
I don't expect it will go that far at all. Surely a company will test the idea of having an AI give executive-level guidance, but when it does so, it will be hastily dismantled. Companies do not structure themselves in the way they do and act the way they do timidly. The C-level executives are not worrying that they could be doing things better. You can see this clearly as essentially every single large company consc…
But what about a new company being ruled by an AI? Surely it would be more efficient and make more money that competitors, eventually displacing them?
Re: Faulty Reward Functions in the Wild
#10This reminds me of how metric-driven companies can go off the rails when they over-optimized for metrics that almost, but not perfectly describe their actual goals.
In theory I imagine the metrics a company defines and attempts to achieve are the ones they want to optimize for. So in essence I can think of only 2 primary reasons it will end badly.
1) They're measuring incorrectly
2) Their "world view" is incorrect
In either case as long as they continually re-assess the above 2, their metrics will change and (ideally) optimize toward a more accurate reflection of reality.
I'd rather have evidence that my world view is incorrect, and learn this as quickly as possible so I can adapt to what the data is telling me.
It's certainly not easy to do this right (and may be impossible to perfect), but I think it's an objectively better option than, say, "going with your gut".