VLA = vision-language-action, a kind of a machine learning model
π0.5: A VLA with open-world generalization
31–40 of 47 posts
Re: π0.5: A VLA with open-world generalization
#32Earlier quoted context omitted.
When you're operating your robot around humans, you want to be very confident it won't injure anyone. It'd be pretty bad if a bug in your code meant instead of putting the cast iron frying pan in the dishwasher, it sent it flying across the room. One way of doing that is to write code with no bugs or unpredictable behaviour, a nigh-impossible feat - especially once you've got ML models in the mix. Another option is t…
Putting the cast iron frying pan in the dishwasher would also be pretty bad.
Maybe this robot is satisfying a rust production utility function. Don't be so bioist. All utility functions are beautiful.
Re: π0.5: A VLA with open-world generalization
#33I'm just a layman, but I can't see this design scaling. It's way too slow and "hard" for fine motor tasks like cleaning up a kitchen or being anywhere around humans, really. I think the future is in "softer" type of robots that can sense whether their robot fingers are pushing a cabinet door (or if it's facing resistance) and adjust accordingly. A quick google search shows this example (animated render) which is clos…
The development here is primarily in the model. If someone invents the 'brains' a robot needs to do useful domestic tasks then there will suddenly be a lot of incentive to build the right body for it.
Right, but ISTM that building the right body is a much harder problem than people are willing to admit.
Software isn't constrained by the harsh truths of physical reality.
Re: π0.5: A VLA with open-world generalization
#34I'm genuinely asking (not trying to be snarky)... Why are these robots so slow? Is it a throughput constraint given too much data from the environment sensors? Is it processing the data? I'm curious about where the bottleneck is.
Not a PI employee, but diffusion policies are like diffusion models for image generation, they generate actions from noise in multiple steps. With current compute you can't run 100+Hz control loops with that kind of architecture. Some combination of distillation, new architectures, faster compute, can eventually attack these problems. Historically as long as something in tech has been shown to be possible, speed has…
That's not the reason, Pi0 was basically predicting at 10hz and predicting a temporal chunk up to 50 points so it could go up to 500Hz.
It's slow because the original telop is slow, and the learned controllers through imitation learning is always a bit slower.
Source : i work on this (not at PI)
Re: π0.5: A VLA with open-world generalization
#35This is amazing! As someone working with industrial robots, normally under strict environmental constraints and control, witnessing such real-world robotics progress truly excites me about the future! By the way, they’ve open-sourced their π0 model (code and model weights). More information can be found here: https://github.com/Physical-Intelligence/openpi
It seems robotics has advanced more in the last 3 years than the previous 20.
the torrent of funding helps here
Re: π0.5: A VLA with open-world generalization
#36I'm genuinely asking (not trying to be snarky)... Why are these robots so slow? Is it a throughput constraint given too much data from the environment sensors? Is it processing the data? I'm curious about where the bottleneck is.
It is inference latency most of the time. These VLA models take in an image + state + text and spit out a set of joint angle deltas.
Depending on the model being used, we may get just one set of joint angle deltas or a series of them. In order to be able to complete a task, it will need to capture images from the cameras, current joint angles and send them to the model along with the task text to get the joint angle changes we will need to apply. Once the joint angles are updated, we will need to check if the task is complete (this can come from the model too). We run this loop till the task is complete.
Combine this with the motion planning that has to happen to make sure the joint angles we are getting do not result in colliding with the surroundings and are safe, results in overall slowness.
Re: π0.5: A VLA with open-world generalization
#37Finally, machines doing the work we dont want to do
https://x.com/ajhai/status/1899528923303809217 something I have been working on for a few months now.
Re: π0.5: A VLA with open-world generalization
#38Earlier quoted context omitted.
When you're operating your robot around humans, you want to be very confident it won't injure anyone. It'd be pretty bad if a bug in your code meant instead of putting the cast iron frying pan in the dishwasher, it sent it flying across the room. One way of doing that is to write code with no bugs or unpredictable behaviour, a nigh-impossible feat - especially once you've got ML models in the mix. Another option is t…
Putting the cast iron frying pan in the dishwasher would also be pretty bad.
Annoying perhaps. But not bad.
Re: π0.5: A VLA with open-world generalization
#39> Investors
> We are grateful for the support of Bond, Jeff Bezos, Khosla Ventures, Lux Capital, OpenAI, Redpoint Ventures, Sequoia Capital, and Thrive Capital.
Re: π0.5: A VLA with open-world generalization
#40Earlier quoted context omitted.
Not a PI employee, but diffusion policies are like diffusion models for image generation, they generate actions from noise in multiple steps. With current compute you can't run 100+Hz control loops with that kind of architecture. Some combination of distillation, new architectures, faster compute, can eventually attack these problems. Historically as long as something in tech has been shown to be possible, speed has…
That's not the reason, Pi0 was basically predicting at 10hz and predicting a temporal chunk up to 50 points so it could go up to 500Hz. It's slow because the original telop is slow, and the learned controllers through imitation learning is always a bit slower. Source : i work on this (not at PI)
Another practical reason is that it's dangerous.
Pi0 uses ARX robot arm which weights 3-4kg per arm. It can easily break things or harm people if you allow it to move fast.