Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a robotics research setup that lives next to my desk

dfdxlabs.com

Post reply on HN
Print view View original

51–60 of 64 posts

Re: Building a robotics research setup that lives next to my desk

#51

Post by b89kim »

- A single arm is sufficient for validating basic Pick/Place tasks, but more complex scenarios require Bi-arm

- Calibration is not required for VLA models.

- RGB or Stereo RGB inputs are sufficient for ACT, DP, and PI0/PI05.

- ROS2 is not strictly required, but it can be useful for sharing/co-developing codes. For instance, the Stanford team built a custom framework for diffusion policy instead. I also developed similar framework because ROS2 is not optimized for bi-manual manipulation or VLA workloads.

Re: Building a robotics research setup that lives next to my desk

#52

Post by sails »

mplappert wrote: post #37
sails wrote: post #35

Earlier quoted context omitted.

Ah no I hadn't seen that, very interesting. I'm finding a gap just before running those experiments. The process of updating firmware, doing basic smoke tests on each device and validating it is live, and can function correctly. Basically the pre-deployment green light that you get on github, but for hardware. Have you seen or thought about that at all?

This seems indeed useful. I haven't seen this for robotics but I'm sure people need this for larger deployments (either for a distributed fleet or for an "arm farm"-like setup where there's many robots in one location for data collection / eval). Interesting idea!

Great thanks for giving it some thought, I’ll share a video as it’s all quite an early concept but saving me a lot of time on the bench!

Re: Building a robotics research setup that lives next to my desk

#53

Post by mplappert »

b89kim wrote: post #51

- A single arm is sufficient for validating basic Pick/Place tasks, but more complex scenarios require Bi-arm - Calibration is not required for VLA models. - RGB or Stereo RGB inputs are sufficient for ACT, DP, and PI0/PI05. - ROS2 is not strictly required, but it can be useful for sharing/co-developing codes. For instance, the Stanford team built a custom framework for diffusion policy instead. I also developed simi…

Did you ever a late RGB vs RGB-D? I haven’t seen that much in the literature.

Also any thoughts on action space representation? Seems to me people are settling on flow matching mostly, but pi still uses discrete tokens to supervise the upstream backbone VLM. I also like the simplicity of discrete bins and used that successfully in the past.

Re: Building a robotics research setup that lives next to my desk

#54

Post by Frannky »

This is so cool and looks so fun. I want to play with robots. I just play with software :(

I wanted to build something making bows automatically and damn, pretty complex and expensive to buy the parts.

I was like, I can probably just buy a 3D printer, print all the parts and buy some motors, but it seems it's way more complicated than that.

Like to play with a single hand robot. It looks like you need 10k+$. I wanted to spend max 1k, 3D print parts, buy motorized parts on Alibaba, and code on my Mac + spare GPUs I have access to. I'll have to save a little :(

Re: Building a robotics research setup that lives next to my desk

#55

Post by delbronski »

Really cool. Good move on writing your own stack. I was weighing using ros2 or building our own, but ended up going the ros2 route to “save time”. We are working with autonomous mobility robots in nature, so we figured the ecosystem around ros2 would be worth the compromise. It was not. All the time we saved setting things up initially, we are paying for now.

There should exist a minimal, clear, robotics library like what you’ve built. The Flask of the robotics world.

Re: Building a robotics research setup that lives next to my desk

#56

Post by b89kim »

mplappert wrote: post #53
b89kim wrote: post #51

- A single arm is sufficient for validating basic Pick/Place tasks, but more complex scenarios require Bi-arm - Calibration is not required for VLA models. - RGB or Stereo RGB inputs are sufficient for ACT, DP, and PI0/PI05. - ROS2 is not strictly required, but it can be useful for sharing/co-developing codes. For instance, the Stanford team built a custom framework for diffusion policy instead. I also developed simi…

Did you ever a late RGB vs RGB-D? I haven’t seen that much in the literature. Also any thoughts on action space representation? Seems to me people are settling on flow matching mostly, but pi still uses discrete tokens to supervise the upstream backbone VLM. I also like the simplicity of discrete bins and used that successfully in the past.

Adding a depth channel rarely yields a massive performance gain, likely due to data scarcity and the fact that modern VLAs are good at guessing distance directly from RGB. I have used multiple RGB-D cameras, but it is hard to get stable images without jitter. Depth can still be useful for high-level reasoning. PI also uses bounding-box or segmentation data from PI-05 for that.

PI smartly combined discretized tokens with flow-matching for efficient training, and it works well in most cases. Still, end-effector representation may be better for teleop with devices like a SpaceMouse, VR, or VibeTracker. PI-07 also supports EEF, but I am not sure how much data is needed to fine-tune PI-05 for that.

I'd suggest starting with the default pi05 model. Data strategy is probably more important than model improvements. Since VLA performance is highly dependent on the data/action distribution and it's easy to modify. After that, you can add high-level reasoning like PI05. I visited a Chinese VLA company that already adopted the PI-05 approach, and it works quite well in practice.

Re: Building a robotics research setup that lives next to my desk

#57

Post by mplappert »

b89kim wrote: post #56
mplappert wrote: post #53

Earlier quoted context omitted.

Did you ever a late RGB vs RGB-D? I haven’t seen that much in the literature. Also any thoughts on action space representation? Seems to me people are settling on flow matching mostly, but pi still uses discrete tokens to supervise the upstream backbone VLM. I also like the simplicity of discrete bins and used that successfully in the past.

Adding a depth channel rarely yields a massive performance gain, likely due to data scarcity and the fact that modern VLAs are good at guessing distance directly from RGB. I have used multiple RGB-D cameras, but it is hard to get stable images without jitter. Depth can still be useful for high-level reasoning. PI also uses bounding-box or segmentation data from PI-05 for that. PI smartly combined discretized tokens w…

This all makes a lot sense, thanks for sharing!

For depth I agree on the VLA route but for ACT / DP-style imitation learning from scratch it seems more feasible (since you’re not fighting a pretrained model that was not trained on this modality). Might also increase robustness since you naturally end up with an input that’s invariant to colors / textures. Plan is to try both paths: the from scratch (and then ablate RGB vs RGB-D) and the VLA + fine-tuning one.

Re: Building a robotics research setup that lives next to my desk

#58

Post by mplappert »

Frannky wrote: post #54

This is so cool and looks so fun. I want to play with robots. I just play with software :( I wanted to build something making bows automatically and damn, pretty complex and expensive to buy the parts. I was like, I can probably just buy a 3D printer, print all the parts and buy some motors, but it seems it's way more complicated than that. Like to play with a single hand robot. It looks like you need 10k+$. I wanted…

I’d recommend looking at the SO-101 then. Much much cheaper.

And hardware is very fun! It’s also very frustrating. But to me worth it.

Re: Building a robotics research setup that lives next to my desk

#59

Post by mplappert »

delbronski wrote: post #55

Really cool. Good move on writing your own stack. I was weighing using ros2 or building our own, but ended up going the ros2 route to “save time”. We are working with autonomous mobility robots in nature, so we figured the ecosystem around ros2 would be worth the compromise. It was not. All the time we saved setting things up initially, we are paying for now. There should exist a minimal, clear, robotics library like…

Yeah I tried to use ROS / Gazebo about 10 years ago (it was still ROS 1) and getting it set up was an immense pain. I remember that creating new modules required writing CMake files. Maybe this is better now, but I decided to skip that.

I also agree on the need for a simple, easy, extensible open source framework. LeRobot IMO is some of this but also contains the dataset + ML code. I think Flask is nice because it's so singularly focused on just one thing with extensibility if you need extras.

So I really like the idea. But being an OSS maintainer these days seems... intense.

Re: Building a robotics research setup that lives next to my desk

#60

Post by b89kim »

mplappert wrote: post #57
b89kim wrote: post #56

Earlier quoted context omitted.

Adding a depth channel rarely yields a massive performance gain, likely due to data scarcity and the fact that modern VLAs are good at guessing distance directly from RGB. I have used multiple RGB-D cameras, but it is hard to get stable images without jitter. Depth can still be useful for high-level reasoning. PI also uses bounding-box or segmentation data from PI-05 for that. PI smartly combined discretized tokens w…

This all makes a lot sense, thanks for sharing! For depth I agree on the VLA route but for ACT / DP-style imitation learning from scratch it seems more feasible (since you’re not fighting a pretrained model that was not trained on this modality). Might also increase robustness since you naturally end up with an input that’s invariant to colors / textures. Plan is to try both paths: the from scratch (and then ablate R…

If you're using depth, you're better off starting with a diffusion policy (DP). We benchmarked ACT, DP, pi0,pi05 on the same task, ACT underperformed in most cases.

There is already plenty of research around multimodal diffusion policies. While DP typically doesn't require pre-training, you can boost data size by depth estimation model+Open data.

Post reply on HN