Finally, machines doing the work we dont want to do
π0.5: A VLA with open-world generalization
21–30 of 47 posts
Re: π0.5: A VLA with open-world generalization
#22These variable-length arrays are getting quite advanced
Precisely my thoughts.
Re: π0.5: A VLA with open-world generalization
#23I'm genuinely asking (not trying to be snarky)... Why are these robots so slow? Is it a throughput constraint given too much data from the environment sensors? Is it processing the data? I'm curious about where the bottleneck is.
The primary bottleneck is typically the motion planning system that must continuously solve complex optimization problems to ensure safe trajectories while avoiding collisions in dynamic environments.
These models typically predict actions directly, there is no motion planning going on here.
Re: π0.5: A VLA with open-world generalization
#24Does the general laws of demos apply here? Than any automation shown is the extent of capabilities not the start?
One thing I notice is that they specify that the robot has never seen the homes before, but certain objects, like the laundry baskets, are identical. Doing your demo is significantly easier if you've already programmed/trained the robot to recognize the specific objects it has to interact with, even if those items are in different locations.
They also got these things working corners of a location instead of stacking tasks on different areas of the same location. And even on these "one-area" task groups it can fail a good amount. Kudos to them for showing the failures though
Re: π0.5: A VLA with open-world generalization
#25Is the robot platform they're using something they've developed themselves? The paper doesn't seem to mention any details outside of sensors and actuators.
[deleted]
Re: π0.5: A VLA with open-world generalization
#26Most of it is open source. Their VLAs are based upon Gemma models + vision encoders, plus their own action experts. You can download and play around or fine tune their Pi0 VLAs from their servers directly (JAX format) or from Huggingface LeRobot safetensors port. They also have notebooks and code in their repo to get started with fine-tuning. Inference runs in a single 4090 RTX streamed over WiFi to the robot.
Re: π0.5: A VLA with open-world generalization
#27I'm genuinely asking (not trying to be snarky)... Why are these robots so slow? Is it a throughput constraint given too much data from the environment sensors? Is it processing the data? I'm curious about where the bottleneck is.
Not a PI employee, but diffusion policies are like diffusion models for image generation, they generate actions from noise in multiple steps. With current compute you can't run 100+Hz control loops with that kind of architecture. Some combination of distillation, new architectures, faster compute, can eventually attack these problems. Historically as long as something in tech has been shown to be possible, speed has…
You're probably right, but for some tasks I suppose you need processing speed, for example bipedal walking.
Re: π0.5: A VLA with open-world generalization
#28Most of it is open source. Their VLAs are based upon Gemma models + vision encoders, plus their own action experts. You can download and play around or fine tune their Pi0 VLAs from their servers directly (JAX format) or from Huggingface LeRobot safetensors port. They also have notebooks and code in their repo to get started with fine-tuning. Inference runs in a single 4090 RTX streamed over WiFi to the robot.
OpenAI is among their investors, which makes me wonder how long their work remains "open".
Re: π0.5: A VLA with open-world generalization
#29Does the general laws of demos apply here? Than any automation shown is the extent of capabilities not the start?
Thank you all I guess the answer is yes.
Re: π0.5: A VLA with open-world generalization
#30I'm genuinely asking (not trying to be snarky)... Why are these robots so slow? Is it a throughput constraint given too much data from the environment sensors? Is it processing the data? I'm curious about where the bottleneck is.
When you're operating your robot around humans, you want to be very confident it won't injure anyone. It'd be pretty bad if a bug in your code meant instead of putting the cast iron frying pan in the dishwasher, it sent it flying across the room. One way of doing that is to write code with no bugs or unpredictable behaviour, a nigh-impossible feat - especially once you've got ML models in the mix. Another option is t…
Putting the cast iron frying pan in the dishwasher would also be pretty bad.