SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
1–10 of 115 posts
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#2[deleted]
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#3>We’ve observed that, throughout the course of training, SIMA 2 agents can perform increasingly complex and new tasks, bootstrapped by trial-and-error and Gemini-based feedback.
>In subsequent training, SIMA 2’s own experience data can then be used to train the next, even more capable version of the agent. We were even able to leverage SIMA 2’s capacity for self-improvement in newly created Genie environments – a major milestone toward training general agents across diverse, generated worlds.
Pretty neat, I wonder how that works with Gemini, I suppose SIMA is a model (agent?) that runs on top of it?
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#4At 0:52 in their demo video, there is a grammatical inconsistency in the agent's text output. The annotations in the video are therefore suspected to be created by humans after the fact. Is Google up to their old marketing/hyping tricks again?
> SIMA 2 Reasoning:
> The user wants me to go to the ‘tomato house’. Based on the description ‘ripe tomato’, I identify the red house down the street.
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#5At 0:52 in their demo video, there is a grammatical inconsistency in the agent's text output. The annotations in the video are therefore suspected to be created by humans after the fact. Is Google up to their old marketing/hyping tricks again? > SIMA 2 Reasoning: > The user wants me to go to the ‘tomato house’. Based on the description ‘ripe tomato’, I identify the red house down the street.
I can't speak to the content of the actual game being played, but it wouldn't surprise me if there was an in-game text prompt:
> "The house that looks like a ripe tomato!"
that was transformed into a "user prompt" in a more instructional format
> "Go to the tomato house"
And both were used in the agent output. At least the Y-axes on the graphs look more reasonable than some other recent benchmarks.
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#6The gap between high level and low level control of robots is closing. Right now thousands of hours of task specific training data is being collected and trained on to create models that can control robots to execute specific tasks in specific contexts. This essentially turns the operation of a robot into a kind of video game, where inputs are only needed a in low-dimensional abstract form, such as "empty the dishwasher" or "repeat what I do" or "put your finger in the loop and pull the string".
This will be combined with high-level control agents like SIMA 2 to create useful real-world robots.
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#7I get why they do it, they are a business. I just wish Google would get off their ivory tower and build in the open more like they used to (did they? maybe I'm misremembering...).
They've acquired this bad habit of keeping all their scientific experiments closed by default and just publishing press releases. I wish it was open-source by default and closed just when there's a good reason.
Don't get me wrong, I suppose this is more of a compliment. I really like what they are doing and I wish we could all participate in these advances.
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#8>We’ve observed that, throughout the course of training, SIMA 2 agents can perform increasingly complex and new tasks, bootstrapped by trial-and-error and Gemini-based feedback. >In subsequent training, SIMA 2’s own experience data can then be used to train the next, even more capable version of the agent. We were even able to leverage SIMA 2’s capacity for self-improvement in newly created Genie environments – a maj…
That’s what it sounded like to me, a plain text interface between two distinct systems.
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#9[flagged]
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#10Isn't most of this demo no man's sky? The voiceover doesn't make it clear that the world is not generated by SIMA.