For solving long term tasks like finding things that aren't there, you can turn the annotated scene into a templated description and feed it to a large-enough model trained on interactive fiction. You are standing in a kitchen. Ahead of you to your right there is a large refrigerator with the handle on the right side. There is a set of cabinets to your left with a plate sitting on the counter above them. > get beer Y…
Great, now we can teach robots to wander around rooms looking for things, saying "keys, keys, keys... where would I put keys?"
It would be really cool if the robot could just know where your keys are by attaching some kind of tile-type thing to it. If it already has a scan of your home, theoretically it could show a photo. But I have no idea if it’s possible to pinpoint an object via rfid.