Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Safety Gym

openai.com

Post reply on HN
Print view View original

11–18 of 18 posts

Re: Safety Gym

#11

Post by peripitea »

Tyr42 wrote: post #9
peripitea wrote: post #3

I'm not super familiar with AI/ML/RL at all, so I'm sure this is a naive question, but isn't it obvious that the answer is to just build in costs to the utility function for behaviors you want to avoid (what they seem to refer to as constrained RL in the article)? That seems both the simplest way to handle it, and most elegant in terms of mapping to the real world domain. Like are there alternate solutions that are e…

I think the answer is that "just building in costs" is actually rather hard to get right. Check out how Concrete Problems in AI Safety (Section 6 in particular is about safe exploration) https://arxiv.org/pdf/1606.06565.pdf Quote: In practice, real world RL projects can often avoid these issues by simply hard-coding an avoidance of catastrophic behaviors. For instance, an RL-based robot helicopter might be programmed…

Yes, that seems like an important problem, but one separate to what they're describing in OP's article. (Again, assuming I'm understanding this right.) Their constrained RL approach is still relying on our ability to enumerate and assign costs to the undesirable behaviors, right? From reading the article, I get the impression that they are focused on addressing that scenario, and leaving the problem of how to enumerate all undesirable behaviors to separate research.

Re: Safety Gym

#12

Post by sanxiyn »

peripitea wrote: post #11
Tyr42 wrote: post #9

Earlier quoted context omitted.

I think the answer is that "just building in costs" is actually rather hard to get right. Check out how Concrete Problems in AI Safety (Section 6 in particular is about safe exploration) https://arxiv.org/pdf/1606.06565.pdf Quote: In practice, real world RL projects can often avoid these issues by simply hard-coding an avoidance of catastrophic behaviors. For instance, an RL-based robot helicopter might be programmed…

Yes, that seems like an important problem, but one separate to what they're describing in OP's article. (Again, assuming I'm understanding this right.) Their constrained RL approach is still relying on our ability to enumerate and assign costs to the undesirable behaviors, right? From reading the article, I get the impression that they are focused on addressing that scenario, and leaving the problem of how to enumera…

Constrained RL is a way to say "thou shalt not murder", instead of saying "murder is utility -10000".

Re: Safety Gym

#14

Post by scottlocklin »

Everything about "openAI" institute seems to be designed to appeal to frightened, paranoid billionaire donors who think they need to be kept safe from near relatives to logistic regression and the remote control for their television, because muh singularity.

Can't you just call it "constrained reinforcement learning" without sexing it up for Elon? I guess not.

Re: Safety Gym

#15

Post by Roark66 »

Tyr42 wrote: post #9
peripitea wrote: post #3

I'm not super familiar with AI/ML/RL at all, so I'm sure this is a naive question, but isn't it obvious that the answer is to just build in costs to the utility function for behaviors you want to avoid (what they seem to refer to as constrained RL in the article)? That seems both the simplest way to handle it, and most elegant in terms of mapping to the real world domain. Like are there alternate solutions that are e…

I think the answer is that "just building in costs" is actually rather hard to get right. Check out how Concrete Problems in AI Safety (Section 6 in particular is about safe exploration) https://arxiv.org/pdf/1606.06565.pdf Quote: In practice, real world RL projects can often avoid these issues by simply hard-coding an avoidance of catastrophic behaviors. For instance, an RL-based robot helicopter might be programmed…

>I think the answer is that "just building in costs" is actually rather hard to get right.

Exactly. It is almost as if we need AI to resolve the problem of properly supervising AI's training. I was wondering if the solution would be to add to classic actor-critic system a third network called a supervisor. The difference between the critic and supervisor would be architecture and the goal of the supervisor would be avoidance of those "terrible" outcomes. Some experiments would have to be run to decide if this approach is viable or do we have to continue tweaking cost functions.

Regarding Safety Gym I'm not sure how what they are doing differs from simply hard coding into your training procedure a series of checks for probability of hitting disallowed states in next step. For example in their example of a robotic arm that is trained with humans around the hard coded algorithm could track people around the arm's work envelope and when some person is detected as approaching it gives the robot a cost penalty. Also, for this to result in trained avoidance of people the network would have to have sufficient inputs to detect people by itself.

Re: Safety Gym

#16

Post by jesseb »

scottlocklin wrote: post #14

Everything about "openAI" institute seems to be designed to appeal to frightened, paranoid billionaire donors who think they need to be kept safe from near relatives to logistic regression and the remote control for their television, because muh singularity. Can't you just call it "constrained reinforcement learning" without sexing it up for Elon? I guess not.

Musk resigned from his seat on the board in 2018. Sam Altman is the current CEO. Not sure what you're getting at other than the usual Musk hate.

Re: Safety Gym

#17

Post by scottlocklin »

jesseb wrote: post #16
scottlocklin wrote: post #14

Everything about "openAI" institute seems to be designed to appeal to frightened, paranoid billionaire donors who think they need to be kept safe from near relatives to logistic regression and the remote control for their television, because muh singularity. Can't you just call it "constrained reinforcement learning" without sexing it up for Elon? I guess not.

Musk resigned from his seat on the board in 2018. Sam Altman is the current CEO. Not sure what you're getting at other than the usual Musk hate.

I like Elon just fine, but OpenAI is basically funded by billionaire donations, and wouldn't exist at all if he hadn't read dumb science fiction masquerading as modern day science fact.

Re: Safety Gym

#18

Post by worik »

scottlocklin wrote: post #17
jesseb wrote: post #16

Earlier quoted context omitted.

Musk resigned from his seat on the board in 2018. Sam Altman is the current CEO. Not sure what you're getting at other than the usual Musk hate.

I like Elon just fine, but OpenAI is basically funded by billionaire donations, and wouldn't exist at all if he hadn't read dumb science fiction masquerading as modern day science fact.

And your problem is???
Post reply on HN