41–50 of 75 posts
There's a cool part where they ask the robot to pick up a lion from a group of toy figures it hasn't seen before. After it does it correctly the NYT reporter asks it to pick up the extinct animal and the robot picks up the dinosaur toy.
Earlier quoted context omitted.
I agree. We need feedback loops on each joint that keep doing what they are doing until a higher feedback loop decides to change things. For example, as I enter this text, most of my fingers are holding the phone and do not need to be told to keep holding it. The low level controllers should also be able to respond to simple things. It's like when I get hurt, my hand yanks back before my brain realizes what has happe…
LLMs have terrible spatial awareness. This probably comes from ONLY being trained on text. I wonder if it would help a LLM running a robot to have a separate controller calculating it's position and what is around it, and feed that into the LLM constantly. It would be like when a video game has a radar display or map to let you know where you are in relation to other things.
You've very much correct that their spatial awareness is terrible. Something as simple as drive forward, then back, turn left, etc. works just fine and they can generally translate it to a specified message format reasonably reliably, but give them something more complex to execute, like drive a robot in a square pattern (an example answer would be go forward, turn right, go forward, turn right, etc.) they start to generate nonsense.
I also tested it with the 30B WizardLM at the time which performed almost as well in terms of message format but had even worse awareness.
Part of the problem is that the training data contains next to no examples that would teach it how 3D space works. I considered making a dataset of driving a robot around with human movement commands and then logging the aggregated sensor data and commands for fine tuning so the prompt format would be pre-learned, but I'm not entirely sure how much it would help.
P.S.: I also was told that the key here is that in automated planning you can't have a human in the loop doing the actual learning. If you are going to prompt engineer or get a human in the loop to a degree that you effectively fool yourself that the robot is solving a problem then its not planning.
[1] https://en.wikipedia.org/wiki/Sussman_anomaly [2] https://chat.openai.com/share/16a8a0e9-7422-41da-a192-6393cc... [3] https://twitter.com/rao2z/status/1599462959788744704?s=20
google demo'd this to an NYT podcast in the final 20 minutes here: https://www.nytimes.com/2023/07/28/podcasts/elons-x-machina-... There's a cool part where they ask the robot to pick up a lion from a group of toy figures it hasn't seen before. After it does it correctly the NYT reporter asks it to pick up the extinct animal and the robot picks up the dinosaur toy.
It would be interesting to see how it solve AI Planning Problems like BlocksWorld[1]. I've read that in the past with these things[2][3] when multiple goals needed to be met at once and there was interaction between them, it just falls over itself. Being able to generate coherent plans and execute them, as I understand, an important aspect of generating Action Sequences given a State and thus for planning in robotics…
[1] https://arxiv.org/abs/2305.15771 [2] https://youtu.be/x10964w00zk
To fully match the human model, AGI based on neural networks will need to dynamically receive information from sensors of all kinds [to be able to sample at least five major categories of physical stimuli, like us]. If we want to reach the level of the neural networks that we carry in our heads, we have to learn more about the role of other brain structures, such as glial cells, which are known to influence the activation and moderation of synapses in the human brain [which correspond to the 'weights' in artificial neural networks], and then apply them to our technological endeavors.
Without it two things will happen
1. We will never have strong AI 2. Strong AI that works virtually only, aware of the world and material reality, but denied access to them by its creators.
Imagine that it is possible to take a snapshot of a human mental state [an analogue of .h5 weights] and then run it on hardware as we do with our neural networks. It would certainly be a nightmare for a mind endowed with "qualia" to find itself enclosed in a metal box [brain in a vat] with no access to the senses. There are clinical cases analogous to this condition, for example in states where the patient is aware of everything around him but is unable to respond.
These issues highlight the ever-present ethical boundaries of IA research. Without integration with real-time sensors AI will always be incomplete and/or inhumane.
This seem like an cool upgrade from RT1, judging from the result. It seems to now also output the delta of the end-effector pose, which was previously handled by a different motion planner. It does seem like this work (and a lot of robot learning works) are still stuck on position/velocity control and not impedance control. Which is essentially output where to go, either closed-loop with a controller or open-loop wit…
1. Choosing a workflow(i.e. a series of general steps that can achieve a goal) 2. Generating low level policy actions given a state(i.e. sensor data).
Like lets say the Workflow is to clean the room. Now there is a big chair in the way and a USB wire there too, the robot could just decide to move around the chair but it could pick up the wire and place it on the table and clean the area. Generating policy actions this way seems to be very much in line with simple VQA based things proposed in the paper, or so it would seem...
It would be interesting to see how it solve AI Planning Problems like BlocksWorld[1]. I've read that in the past with these things[2][3] when multiple goals needed to be met at once and there was interaction between them, it just falls over itself. Being able to generate coherent plans and execute them, as I understand, an important aspect of generating Action Sequences given a State and thus for planning in robotics…
Related work is the planning paper by Valmeekan et al [1]. The gist is that LLMs are incapable of planning, which is due to their autoregressive nature. METAs Head of AI Yann Lecun also talks about this topic in a talk [2]. As RT2 is based on a similar architecture, I think the results will be similar. [1] https://arxiv.org/abs/2305.15771 [2] https://youtu.be/x10964w00zk