111–120 of 178 posts
Until we get robots with really good hands, something I'd love in the interim is a system that uses _me_ as the hands. When it's time to put groceries away, I don't want to have to think about how to organize everything. Just figure out which grocery items I have, what storage I have available, come up with an optimized organization solution, then tell me where to put things, one at a time. I'm cautiously optimistic…
Goal 2 has been achieved, at least as a proof of concept (and not by OpenAI): https://openai.com/index/openai-technical-goals/
They can put away clutter but if they could chop a carrot or dust a vase they'd have shown videos demonstrating that sort of capability. EDIT: Let alone chop an onion. Let me tell you having a robot manipulate onions is the worst. Dealing with loose onion skins is very hard.
It seems that end to end neural networks for robotics are really taking off. Can someone point me towards where to learn about these, what the state of the art architectures look like, etc? Do they just convert the video into a stream of tokens, run it through a transformer, and output a stream of tokens?
I'd be very interested to see what the output of their 'big model' is that feeds into the small model. I presume the small model gets a bunch of environmental input, and some input from the big model, and we know that the big model input only updates every 30 or 40 frames in terms of small model.
Like, do they just output random control tokens from big model and embed those in small model and do gradient descent to find a good control 'language'? Do they train the small model on english tokens and have the big model output those? Custom coordinates tokens? (probably). Lots of interesting possibilities here.
By the way, the dataset they describe was generated by a large (much larger presumably) vision model tasked with creating tasks from successful videos.
So the pipeline is:
* Video of robot doing something
* (o1 or some other high end model) "describe very precisely the task the robot was given"
* o1 output -> 7B model -> small model -> loss
That’s how I feel about LLMs and code.
Earlier quoted context omitted.
Yeah, but a robot will work 24/7, not 2h byweekly -_-
do you need a robot to work in your house 24/7?
The article mentions that the system in each robot uses two ai models.
S2 is built on a 7B-parameter open-source, open-weight VLM pretrained on internet-scale data
S1, an 80M parameter cross-attention encoder-decoder transformer, handles low-level [motor?] control.
What part of this system understands 3 dimensional space of that kitchen?
How does the robot closest to the refrigerator know to pass the cookies to the robot on the left?
How is this kind of speech to text, visual identification, decision making, motor control, multi-robot coordination and navigation of 3d space possible locally?
Figure robots, each equipped with dual low-power-consumption embedded GPUs
Earlier quoted context omitted.
But what if there's no YouTube tutorial for the exact AC unit you have and it doesn't look like any of the videos you checked out?
Then you are equally fucked as the AI will be, so no difference. Case in point, I remember about ten years ago our washing machine started making noise from the drum bearing. Found a Youtube tutorial for bearing replacement on the exact same model, but 3 years older. Followed it just fine until it was time to split the drum. Then it turned out that in the newer units like mine, some rent-seeking MBA fuckers had decid…
Why do they make “eye contact” after every hand off? Feels oddly forced.
Are we at a point now where Asimov’s laws are programmed into these fellas somewhere?
The article clearly spells out that it's end to end LLM. Text and video in, motor function out.
Technically, the text model probably has a few copies, but they are nothing more than Asimov's narrative. Laws don't (and can't) exist in a model
I don't know, there has been so many overhyped and faked demos in humanoid robotics space over the last couple years, it is difficult to believe what is clearly a demo release for shareholders. Would love to see some demonstration in a less controlled environment.
Imagine they bring one out to a construction site and they treat the robot as a new rookie guy, go pick up those pipes. That would be an ultimate on the fly test to me.