Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI agents that “self-reflect” perform better in changing environments

hai.stanford.edu

Post reply on HN
Print view View original

11–20 of 46 posts

Re: AI agents that “self-reflect” perform better in changing environments

#11

Post by neuronerd1 »

ftxbro wrote: post #2

So from this hacker news title I definitely thought it was saying that when you give some AI agents a self reflection like maybe by putting an internal monologue loop then they unlock an emergent animal-like exploration behavior. But this is not what happened. Instead, some guys told AI agents to explore in the way that the guys think that animals explore. "Stanford researchers invented the “curious replay” training…

Author here, a key thing is that we didn't prescribe that the mechanism of exploration was the same, but rather we found that the AI agent explored poorly (i.e. unlike animals) until we included Curious Replay. Interestingly, we found that the benefits of Curious Replay also led to state of the art performance on Crafter.

Re: AI agents that “self-reflect” perform better in changing environments

#12

Post by ftxbro »

neuronerd1 wrote: post #11
ftxbro wrote: post #2

So from this hacker news title I definitely thought it was saying that when you give some AI agents a self reflection like maybe by putting an internal monologue loop then they unlock an emergent animal-like exploration behavior. But this is not what happened. Instead, some guys told AI agents to explore in the way that the guys think that animals explore. "Stanford researchers invented the “curious replay” training…

Author here, a key thing is that we didn't prescribe that the mechanism of exploration was the same, but rather we found that the AI agent explored poorly (i.e. unlike animals) until we included Curious Replay. Interestingly, we found that the benefits of Curious Replay also led to state of the art performance on Crafter.

OK here is the arxiv https://arxiv.org/abs/2306.15934 called "Curious Replay for Model-based Adaptation" and from the abstract it says "we present Curious Replay -- a form of prioritized experience replay tailored to model-based agents through use of a curiosity-based priority signal" and "DreamerV3 with Curious Replay surpasses state-of-the-art performance on Crafter" here is the crafter benchmark https://github.com/danijar/crafter but it appears to have out of date baselines at the bottom of that page.

That arxiv stuff looks perfectly normal but I kind of hate how it got more and more caricatured as it went through the university press office and hacker news clickbait pipeline.

Re: AI agents that “self-reflect” perform better in changing environments

#14

Post by kromem »

neuronerd1 wrote: post #11
ftxbro wrote: post #2

So from this hacker news title I definitely thought it was saying that when you give some AI agents a self reflection like maybe by putting an internal monologue loop then they unlock an emergent animal-like exploration behavior. But this is not what happened. Instead, some guys told AI agents to explore in the way that the guys think that animals explore. "Stanford researchers invented the “curious replay” training…

Author here, a key thing is that we didn't prescribe that the mechanism of exploration was the same, but rather we found that the AI agent explored poorly (i.e. unlike animals) until we included Curious Replay. Interestingly, we found that the benefits of Curious Replay also led to state of the art performance on Crafter.

It's very cool work.

I've been wondering for a while at what the next steps in adding 'inefficiencies' to AI processing would look like, commenting the other day to a friend that what's needed in the next 18 months is getting AI to be able to replicate the Eureka moments in the shower where latent information is reconstructed in parallel to processing tangential topics.

Going from "attention is all you need" to "attention and curiosity is what you need" seems like a great next step!

Re: AI agents that “self-reflect” perform better in changing environments

#15

Post by lcnPylGDnU4H9OF »

ShamelessC wrote: post #9
ftxbro wrote: post #2

So from this hacker news title I definitely thought it was saying that when you give some AI agents a self reflection like maybe by putting an internal monologue loop then they unlock an emergent animal-like exploration behavior. But this is not what happened. Instead, some guys told AI agents to explore in the way that the guys think that animals explore. "Stanford researchers invented the “curious replay” training…

I hate that titles can differ from the article here. It’s patronizing and commonly inaccurate or misleading.

“Patronizing” seems to be a matter of taste. I’ve never considered it to be patronizing; indeed, that’s often much unlike articles which have their title changed.

As far as simply differing, much of the time there’s a character limit that’s hit. I’ve seen many posts with comment from the poster calling out their edit to the title and the character limit is usually cited.

It would be especially difficult to keep the character limit (I think there are legitimate design reasons for this) while also requiring that the title matches the submission as closely as possible. Who decides what words are omitted without it potentially being any of: patronizing, inaccurate, or misleading?

Re: AI agents that “self-reflect” perform better in changing environments

#16

Post by ftxbro »

ShamelessC wrote: post #9
ftxbro wrote: post #2

So from this hacker news title I definitely thought it was saying that when you give some AI agents a self reflection like maybe by putting an internal monologue loop then they unlock an emergent animal-like exploration behavior. But this is not what happened. Instead, some guys told AI agents to explore in the way that the guys think that animals explore. "Stanford researchers invented the “curious replay” training…

I hate that titles can differ from the article here. It’s patronizing and commonly inaccurate or misleading.

I don't like the misleading titles either, but honestly if you want the real titles you probably want some kind of arxiv feed. The paper title is "Curious Replay for Model-based Adaptation" which is too dry for social media or whatever hacker news is or for whoever is the audience of the stanford university press office. You have to expect more juicy (and therefore somewhat misleading or sensationalized) titles if you don't get your news straight from an arxiv feed.

Re: AI agents that “self-reflect” perform better in changing environments

#17

Post by mandmandam »

Schnitzkitz wrote: post #13

Makes sense. AI lacks rationality, and animals lack rationality. Of course, humans are the rational animal, and hence we know when we truly understand things or when we just repeat or spitball.

Nah, not really.

History has been repeating itself for thousands of years. We keep killing the prophets, and putting the absolute worst of us on pedestals. What's rational about that?

Dolphins mucking about in the water - that's rational.

Re: AI agents that “self-reflect” perform better in changing environments

#18

Post by hirundo »

neuronerd1 wrote: post #11
ftxbro wrote: post #2

So from this hacker news title I definitely thought it was saying that when you give some AI agents a self reflection like maybe by putting an internal monologue loop then they unlock an emergent animal-like exploration behavior. But this is not what happened. Instead, some guys told AI agents to explore in the way that the guys think that animals explore. "Stanford researchers invented the “curious replay” training…

Author here, a key thing is that we didn't prescribe that the mechanism of exploration was the same, but rather we found that the AI agent explored poorly (i.e. unlike animals) until we included Curious Replay. Interestingly, we found that the benefits of Curious Replay also led to state of the art performance on Crafter.

Is there a possible Crafter benchmark that is too high for safety? For instance, a number beyond which it would be dangerous to release a well equipped agent into meatspace with the goal of maximizing paperclips?

Re: AI agents that “self-reflect” perform better in changing environments

#19

Post by izzygonzalez »

hirundo wrote: post #18
neuronerd1 wrote: post #11

Earlier quoted context omitted.

Author here, a key thing is that we didn't prescribe that the mechanism of exploration was the same, but rather we found that the AI agent explored poorly (i.e. unlike animals) until we included Curious Replay. Interestingly, we found that the benefits of Curious Replay also led to state of the art performance on Crafter.

Is there a possible Crafter benchmark that is too high for safety? For instance, a number beyond which it would be dangerous to release a well equipped agent into meatspace with the goal of maximizing paperclips?

This is absurd.

Re: AI agents that “self-reflect” perform better in changing environments

#20

Post by ftxbro »

hirundo wrote: post #18
neuronerd1 wrote: post #11

Earlier quoted context omitted.

Author here, a key thing is that we didn't prescribe that the mechanism of exploration was the same, but rather we found that the AI agent explored poorly (i.e. unlike animals) until we included Curious Replay. Interestingly, we found that the benefits of Curious Replay also led to state of the art performance on Crafter.

Is there a possible Crafter benchmark that is too high for safety? For instance, a number beyond which it would be dangerous to release a well equipped agent into meatspace with the goal of maximizing paperclips?

human level is about 50 and as long as they don't allow to craft paperclips i think it's ok
Post reply on HN