Gemini Robotics 2 brings whole body intelligence to robots

I don't think that there's any inherent difficulty to do with physical bodies, it's just that the learning techniques used by the big labs, that is to say, various forms of Reinforcement Learning, don't transfer to the real world. Because in the real world if you try stuff at random until you figure out the right policy, you end up destroying too many robot bodies. Imitation learning that went wild a couple years ago with ALOHA looks like a great way to get around this on paper, but then you realise that you have to train your robot for hours on every single task you want it to perform and it still won't generalise even between similar tasks. The sim-2-real gap also remains a big obstacle and you can't just train in simulation and transfer to the real world seamlessly; and then the seams end up being more like chasms.

Then there's the problem of generalisation. RL is insane in figuring out the dynamics of any arbitrary environment you drop it in at any time. Unfortunately once you have a policy trained in one environment (or one task) you have to train again for the next environment (or task) you want to deal with. The number of different environments, situations or tasks in the real world ends up being overwhelmingly large.

So for example DeepMind has been trying for ever to train various robots to do stuff like grasp arbitrary objects by training in the real world or in simulation and still grasping is an unsolved problem, in the general case; and in fact in most cases.

With LLMs it was different in that there was a www of text to train on and the stakes are lower because there's no sim-2-real gap and there's no need to deal with the real world, in training or deployment, it's just text on a screen. That's very hard to replicate in robotics.

There are major breakthroughs still to be made and they aren't currently even on the horizon. And no, I don't have the answer with my super secrete robotic AGI homebrew project either, just saying :P

No, the above is absolutely exactly true. Robots folding clothes and tying shoelaces etc is nothing but a tech demo at this point.

It's difficult to grok this because if you watch a human folding a t-shirt, you can reliably predict that the same human will fold a different t-shirt just as well, and in fact be perfectly capable of folding a wide variety of other clothes items as well. Not so for robots. With robots, what you see is precisely what you get. If you see a robot folding a t-shirt, all that means is that that particular robot can fold that particular t-shirt. The state of the art today is that the same robot cannot be expected to be able to fold a different t-shirt.

For example, see this article about Mobile ALOHA at Google. There's a passage where the visiting, awe-struck, journalist asks whether the robot he's just seen tying up a pair of shoelaces can tie up his own shoe.

“If I gave it my shoe,” I ventured, “would it just totally fail?”

“We could try,” Tompson said. I removed my right sneaker, with apologies to anyone forced to handle it. Tompson gamely placed it on the table, while Driess reloaded the policy.

“To set expectations,” Driess said, “this is a task that is thought of as being impossible.”

Tompson eyed his new experimental subject with some trepidation. “Very short shoelaces,” he said.

The policy booted up, and the claws set to work. This time, they poked at the shoelace without getting a grip. “Do you give consent for your shoe to be destroyed?” Driess joked, as the hands grabbed at the tongue. Tompson let them try for a few more seconds before hitting the Failure pedal.

https://archive.ph/CiJJG#selection-1887.0-1915.291

(Original: https://www.newyorker.com/magazine/2024/12/02/a-revolution-i...)

As to ACT-2 which basically uses the same techniques as ALOHA (imitation learning) far as I can tell, that's a commercial product and the information they give on their site is difficult to parse. E.g. they say they have 99.1% ±0.3 success rate, 778 successful folds and 9 garment types which is low enough to engender some trust they're not trying to inflate their numbers, but they don't say whether they trained on the garments used in evaluation or not. Chances are they did, because that's the current limit of the technology, i.e. if the garment being folded is unseen (as opposed to the environment, which they tout) then performance is basically random. So either ACT-2 have a major breakthrough that is a few leaps and bounds away from the current state of the art, or you've just watched a tech demo.

The fact that they only advertise "9 garment types" though is a big hint: they have the same problem with generalisation as everybody else at this point in time.

Yeah, look carefully at that demo of tying the garbage bag strings: it's done in a very peculiar style that suggests a very specific, very precise, "algorithm" taught in an imitation learning session, which has no chance to transfer to other tasks, or even other garbage bag strings.

As usual with robot tech demos: WYSIWYG.

Depends entirely on VLA arch. Some have dedicated action diffusion heads that work in a standalone non-text action output space. Much like an LLM can either use an external TTS or have audio output heads attached to it directly for native S2S.

But your entire premise is wrong regardless of that.

Even if VLAs were forever bound to outputting text, you'd have to prove that they're fundamentally incapable of emitting text that maps to useful action sequences. No proof of that whatsoever - and plenty of empirical evidence suggests otherwise. Even non-specialist LLMs like ChatGPT are getting better at controlling robots and navigating 3D environments, if slowly.

you mean like the self driving car which elon promised ten years ago. :D Robotic vision AI is far more complex than an LLM