Isn't most of this demo no man's sky? The voiceover doesn't make it clear that the world is not generated by SIMA.
This is not a world generating model.
It is a game playing model.
SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
31–40 of 115 posts
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#32I hope we can get some (ideally local) version of this we can use as a "gaming minion". There's a lot of games where I probably would have played more if I could delegate the grind. If they're not that competent, it adds to the fun a little even.
sorry this is kind of nuts to me. You want something to play video games for you because the video game isn't fun? Just play a game that is fun. The point of the game is to play it
I mean, that's literally a RTS?
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#33I get why they do it, they are a business. I just wish Google would get off their ivory tower and build in the open more like they used to (did they? maybe I'm misremembering...). They've acquired this bad habit of keeping all their scientific experiments closed by default and just publishing press releases. I wish it was open-source by default and closed just when there's a good reason. Don't get me wrong, I suppose…
Same! I want to play with this so bad!
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#34This is obviously just a research project, but I do wonder about the next steps: * After exploring an learning about a virtual world, can anything at all be transferred to an agent operating in the real world? Or would an agent operating in the real world have to be trained exclusively or partially in the real world? * These virtual worlds are obviously limited in a lot of important ways (for example, character locom…
Look into the sim2real problem in robotics
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#35Earlier quoted context omitted.
To be fair, it's 65% on all tasks (with a 75% human baseline) and 15% on unseen environments. They don't provide a human baseline for that, but I'd imagine it's much more than 15%.
It really feels like we are determined to simulate every possible task in every possible environment instead of building true intelligence.
I'm curious what's your definition of "true intelligence"
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#36Isn't most of this demo no man's sky? The voiceover doesn't make it clear that the world is not generated by SIMA.
This is not a world generating model. It is a game playing model.
And my post is saying that if you don't really know better, from the narration, you'd think google also generated the world. At least that was my impression, and I'm vaguely familiar with these things.
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#37The gap between high level and low level control of robots is closing. Right now thousands of hours of task specific training data is being collected and trained on to create models that can control robots to execute specific tasks in specific contexts. This essentially turns the operation of a robot into a kind of video game, where inputs are only needed a in low-dimensional abstract form, such as "empty the dishwas…
I work on a much easier problem (physics-based character animation) after spending a few years in motion planning, and I haven’t really seen anything to suggest that the problem is going to be solved any time soon by collecting more data.
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#38I hope we can get some (ideally local) version of this we can use as a "gaming minion". There's a lot of games where I probably would have played more if I could delegate the grind. If they're not that competent, it adds to the fun a little even.
One thing I do with games is automate the grind. To me, that is part of the fun. I have built lego robots to press a sequence of buttons repeatedly, or programmed microcontrollers using circuitpython to press a series of keys or click the mouse at given intervals to grind various in-game currency and such. It's so common for me to do these kinds of things that I now instinctively look for places in gameplay that I can automate. I haven't done anything as complicated as using computer vision to look at the screen and respond to it, but I did see that Anthony Sottile did this to catch shiny pokemon https://youtu.be/-0GIY5Ixgkk and doing something like this has been out there on my horizon.
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#39At 0:52 in their demo video, there is a grammatical inconsistency in the agent's text output. The annotations in the video are therefore suspected to be created by humans after the fact. Is Google up to their old marketing/hyping tricks again? > SIMA 2 Reasoning: > The user wants me to go to the ‘tomato house’. Based on the description ‘ripe tomato’, I identify the red house down the street.
The scene just before you describe has the user write "ripe tomato" in the description - you can see it in the video. The summary elides it, but the "ripe tomato" instruction is also clearly part of the context.
Re: SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
#40OK, AI playing video games is cool. But you know what's really really cool? It looks like SIMA 2 is controlling the mouse and reading the screen at something approaching 30+fps. WANT. Computer use agents are so slow right now, this is really something. I wonder what the architecture is for this.