Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Figure 01 robot demos its OpenAI integration

twitter.com

Post reply on HN
Print view View original

41–50 of 116 posts

Re: Figure 01 robot demos its OpenAI integration

#41

Post by golol »

lelag wrote: post #27

It's a really cool demo and I'm impressed by the dexterity of the robot, however I'm a bit underwhelm by what's shown here in the sense that the speech and reasoning capabilities is just obvious to anyone who's been paying attention and has experience with GPT4. The function calling was great, but it had a very simple "world" to interact with. It's really interesting to see it integrated with a robot that can interac…

Yea of course this is just an LLM interacting through a very crude interface with some control algorithms, but I think it is amazing that we sort of have an approach for both ends of the complexity spectrum now: LLMs for high level, vague, common sense reasoning, and traditional robot control, planning, machine learning methods for the physicak execution of simple movements. We just have to gradually connect these two systems.

Re: Figure 01 robot demos its OpenAI integration

#43

Post by margorczynski »

tomcam wrote: post #28
modeless wrote: post #13

Earlier quoted context omitted.

That delay will be eliminated very soon. IMO low latency natural voice conversations are going to be bigger than ChatGPT. It's going to blow people's minds when they can converse with these AIs casually just like with their real life friends. It won't be anything like Siri or Alexa anymore. Here's a demo from a startup in this space. Still very early. https://deck.sindarin.tech/

I’ll be thrilled when Siri can spell my wife’s name correctly after 13 years of continuous usage and explicitly training it in her name. Admittedly her name is wildly complicated and totally unknown to the software folks at Apple: Ada Also my radio-trained voice is so generic a caller every week-ish assumes I am a bot, so I’m pretty sure the problem isn’t me enunciation or accent.

LLMs can't really spell as the smallest "block" of information they operate on are tokenized whole words. Same issue with e.g. arithmetic.

Re: Figure 01 robot demos its OpenAI integration

#44

Post by margorczynski »

famouswaffles wrote: post #34
huytersd wrote: post #32

Earlier quoted context omitted.

How though. It’s probably just predefined actions that are triggered by the LLM output. At the same time it would be impressive if the LLM determined the right function to call in real time, was able to deal with the ambiguity in placement of the garbage and bonus points if it could do that in a scenario that wasn’t hardcoded to exactly standing behind that table in that spot.

The robot movements are separate end-to-end neural networks. They're triggered by the LLM but aren't hardcoded.

Why does it then after performing an action or whole plan return to the "default" position with the hands in that strange stance? Looks kinda like there is some "hard coded" flow that simply uses the LLM(s) to perform actions.

Re: Figure 01 robot demos its OpenAI integration

#45

Post by gfodor »

eigenvalue wrote: post #11

This shows the real utility of the Groq low latency inference. That delay in responding makes it much less impressive (it’s still really impressive obviously!)

The bottleneck here is the multimodal vision processing, at least if my experience building this kind of thing is any indication. Afaik Groq has not demonstrated the speeds they have for this. (Obviously they'll be better than OpenAI, but it still may be slow enough to leave people disappointed.)

Re: Figure 01 robot demos its OpenAI integration

#46

Post by modeless »

Selecting one of a set of pre-trained actions by voice is cool but not exactly ground-breaking. Using GPT-4V to describe a scene is also pretty simple. The most impressive things here to me are the speed of picking up the trash and the fluid passing of objects between hands.

It's unclear how general these movement policies are though. The way that guy is standing perfectly still makes me think that it would fail if everything wasn't set up just so. I'd like to see demos with more variation.

I don't want to be too negative here though. I think it's a great demo and I can't wait to see more.

Re: Figure 01 robot demos its OpenAI integration

#47

Post by ta8645 »

margorczynski wrote: post #43
tomcam wrote: post #28

Earlier quoted context omitted.

I’ll be thrilled when Siri can spell my wife’s name correctly after 13 years of continuous usage and explicitly training it in her name. Admittedly her name is wildly complicated and totally unknown to the software folks at Apple: Ada Also my radio-trained voice is so generic a caller every week-ish assumes I am a bot, so I’m pretty sure the problem isn’t me enunciation or accent.

LLMs can't really spell as the smallest "block" of information they operate on are tokenized whole words. Same issue with e.g. arithmetic.

But they should be able to select the correct token for homophones, which amounts to the same thing.

Re: Figure 01 robot demos its OpenAI integration

#48

Post by mvkel »

archermarks wrote: post #25
jaffee wrote: post #5

Earlier quoted context omitted.

> text to servo movement yeah this was super impressive. If this is at the point where you can put an arbitrary object in front of it and ask it to move it somewhere, that's going to be huge for industrial automation type stuff I'd imagine. I do wonder how much of that demo was pre-baked/trained though. Could they repeat the same thing with a banana? What if the table was more cluttered? What if there were two people…

Knowing as many people in the robotics space as I do, I suspect the demo may not be completely "pre-baked" but it is almost certaintly highly selected. Often they'll try the demo many many times until they get a clean run-through without mistakes. The circumstances are also likely pretty idealized, like they pick objects and settings that they know it performs well in.

Interesting! This would sort of explain the low energy of the human demonstrator in the video

"Take 488... action!"

Re: Figure 01 robot demos its OpenAI integration

#49

Post by huytersd »

famouswaffles wrote: post #34
huytersd wrote: post #32

Earlier quoted context omitted.

How though. It’s probably just predefined actions that are triggered by the LLM output. At the same time it would be impressive if the LLM determined the right function to call in real time, was able to deal with the ambiguity in placement of the garbage and bonus points if it could do that in a scenario that wasn’t hardcoded to exactly standing behind that table in that spot.

The robot movements are separate end-to-end neural networks. They're triggered by the LLM but aren't hardcoded.

Do you know this for a fact or are you speculating?

Re: Figure 01 robot demos its OpenAI integration

#50

Post by bigyikes »

m3kw9 wrote: post #36
daveguy wrote: post #14

The speech to servo movement is impressive as others pointed out. What strikes me as amazing is the speed with which it is performing tasks that require dexterity. This is the first object manipulation robot demo I have seen that didn't require speeding up the video for it to look "natural".

Is it just me or are people easily impressed by these robot demo that moves nothing like humans and doing the simplistic tasks like passing a large object. The freaking Apple is within his arms reach and he asked to pass the Apple. I almost laughed out loud.

Easily impressed? These are difficult tasks! Even being able to grip the apple without crushing it is impressive.

Not sure anything here is state of the art, but that doesn’t make it easy.

Post reply on HN