Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Helix: A vision-language-action model for generalist humanoid control

figure.ai

Post reply on HN
Print view View original

11–20 of 178 posts

Re: Helix: A vision-language-action model for generalist humanoid control

#11

Post by jayd16 »

traverseda wrote: post #5

"The first time you've seen these objects" is a weird thing to say. One presumes that this is already in their training set, and that these models aren't storing a huge amount of data in their context, so what does that even mean?

It probably gives them confidence that they can accurately see a thing even though they don't know what that thing is.

I could also imagine a lot of safety around leaving things outside of the current task alone so you might have to bend over backwards to get new objects worked on.

Re: Helix: A vision-language-action model for generalist humanoid control

#15

Post by wwwtyro »

Until we get robots with really good hands, something I'd love in the interim is a system that uses _me_ as the hands. When it's time to put groceries away, I don't want to have to think about how to organize everything. Just figure out which grocery items I have, what storage I have available, come up with an optimized organization solution, then tell me where to put things, one at a time. I'm cautiously optimistic this will be doable in the near term with a combination of AR and AI.

Re: Helix: A vision-language-action model for generalist humanoid control

#18

Post by Symmetry »

So, there's no way you can have fully actuated control of every finger joint with just 35 degrees of freedom. Which is very reasonable! Humans can't individually control each of our finger joints either. But I'm curious how their hand setups work, which parts are actuated and which are compliant. In the videos I'm not seeing any in-hand manipulation other than just grasping, releasing, and maintaining the orientation of the object relative to the hand and I'm curious how much it can do / they plan to have it be able to do. Do they have any plans to try to mimic OpenAI's one handed rubics cube demo?

Re: Helix: A vision-language-action model for generalist humanoid control

#19

Post by NitpickLawyer »

Philpax wrote: post #10
bhouston wrote: post #4

When doing robot control, how do you model in the control of the robot? Do you have tool_use / function calling at the top level model which then gets turned into motion control parameters via inverse kinematic controllers? What is the interface from the top level to the motors? I feel it can not just be a neural network all the way down, right?

Have a look at the post - it explains how it works. There are two models: a 7-9Hz 7B vision-language model, and a 200Hz 80M visuomotor model. The former produces a latent vector, which is then interpreted by the latter to drive the motors.

> a 7-9Hz 7B vision-language model, and a 200Hz 80M visuomotor model.

huh. An interesting approach. I wonder if something like this can be used for other things as well, like "computer use" with the same concept of a "large" model handling the goals, and a "small" model handling clicking and stuff, at much higher rates, useful for games and things like that.

Re: Helix: A vision-language-action model for generalist humanoid control

#20

Post by Symmetry »

traverseda wrote: post #5

"The first time you've seen these objects" is a weird thing to say. One presumes that this is already in their training set, and that these models aren't storing a huge amount of data in their context, so what does that even mean?

It's normal to have a training set and a validation set and I interpreted that to mean that these items weren't in the training set.
Post reply on HN