Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Untitled topic

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 75 posts

Re: undefined

#61

Post by whinenot »

NalNezumi wrote: post #33
empath-nirvana wrote: post #26

Earlier quoted context omitted.

Seems to me that the way to architect this is to have multiple quasi independent embedded controllers at different levels. For example, you might have 3 independent finger controllers, managed by a hand controller, and so on, on up to the highest level LLM that drives everything. So you have an LLM that just says, pick up the green can, issues whatever structured data needs to go to the arm controller and on down the…

That is indeed what a lot of Machine Learning turned Robotics researchers/enthusiasts are banking on. A counter argument to that is what Dhruv Batra responded to the question "lol why not use LLM for everything" [1]. >As a human, I don't understand the detailed micro-second by microsecond movements my fingers have to do to pick something up, let alone how I'm touch-typing this sentence. It just sort of "happens" when…

>It's harder to reverse-engineer an unconscious process

Aside from some basic life support systems, don't almost all movements start with conscious effort? Whether you are deliberate about the exercise or not, you practice and practice until you develop 'muscle memory' where it becomes unconscious: walking, dribbling a basketball, holding a G chord on a guitar, etc.

Re: undefined

#62

Post by blovescoffee »

NalNezumi wrote: post #34
blovescoffee wrote: post #31

Earlier quoted context omitted.

They collect data for RL/IL in simulation which _can_ generalize to the real world. Also, being Google, they have the resources to collect data by brute force i.e. scientists manually collecting that data. The paper says - one main source of the data is internet scale vision/llm data. The second source is 6k trial runs. A principal idea behind this work is that you can collect data in one domain to avoid collecting d…

They use the RT1 data which is (if I remember correctly) 11 month of data using like 6 research engineers sitting everyday, remote controlling the robot. They might even use ROSIE data which is data-augmentation using Stable-diffusion inpainting. And for that they can do mobile manipulation, which have been possible for a long while (as long as we know ground-truth of object location, and have a some-what consistent…

Sure, I wouldn't say reasoning can guide a robot to do gymnastics zero-shot either. But it is a substantiated approach to generalizing existing capabilities - just like in humans. You or a robot can't reason your way to bike-riding but if you know how to bake cookies or brush your teeth, you can perform much better zero-shot on related tasks like baking a chicken or brushing a mirror.

Re: undefined

#63

Post by tnecniv »

amelius wrote: post #53
neatze wrote: post #8

Earlier quoted context omitted.

This is false to very large degree, you have simulations for robotics, and you can buy/build your self cheaper versions.

What are some good open source simulators suitable for robotics development? (preferably a physics simulator library, not an entire framework)

PyBullet (I guess Bullet more broadly) sees a lot of use

Re: undefined

#64

Post by taneq »

ChatGTP wrote: post #22
taneq wrote: post #5

The Portal-themed animation really sells it as a step in the right direction. ;) Seriously though, I’m excited to see where this goes. AI is progressing so fast now that newcomers loudly proclaim “AI is dead” and “going nowhere” when they haven’t seen SotA beaten for a whole month, because they don’t remember the times when a small improvement in a decade was big news.

You know where it's going dude...to the military and law enforcement. It's completely naive to think otherwise. Advanced robotics might be used to unpack your dishwasher, or save kittens stuck in trees, but we know where the $$$ lies...

Depends on who sparks the hard takeoff. There's a whole bunch of interesting times coming up and I don't think our human power structures are going to have any control over it. The next five years are going to be less about status quo and more about... good parenting?

Re: undefined

#66

Post by [deleted] »

peterleiser wrote: post #65
darkclouds wrote: post #36

Sounds cool, any youtube evidence?

The post links to: https://robotics-transformer2.github.io/ There under "demo" are a few videos at 2x and 4x speed. It's slow. None of the videos include audio of the verbal commands or the latency between commands and action.

[deleted]

Re: undefined

#67

Post by StackOverlord »

empath-nirvana wrote: post #26
NalNezumi wrote: post #19

This seem like an cool upgrade from RT1, judging from the result. It seems to now also output the delta of the end-effector pose, which was previously handled by a different motion planner. It does seem like this work (and a lot of robot learning works) are still stuck on position/velocity control and not impedance control. Which is essentially output where to go, either closed-loop with a controller or open-loop wit…

Seems to me that the way to architect this is to have multiple quasi independent embedded controllers at different levels. For example, you might have 3 independent finger controllers, managed by a hand controller, and so on, on up to the highest level LLM that drives everything. So you have an LLM that just says, pick up the green can, issues whatever structured data needs to go to the arm controller and on down the…

This is in fact what happens in the human body.

For example, when you reach out to pick up a green can, your brain makes the decision to do the task but it's your spinal cord and peripheral nerves that carry out the detailed work – orienting the hand, managing grasp strength, controlling the arm movements etc. This process is mostly unconscious – you don't need to actively think about how to tense each muscle in the same way that an embedded controller wouldn't need to understanding the working of the entire robotic system to carry out its specific task.

Much like the model suggested, the human body communicates feedback across layers — this process is crucial to maintaining balance, coordination and effectively reacting to the environment. For instance, if your fingers touch a hot stove, the sensory receptors in your skin will immediately send a signal to your spinal cord and a reflex action will make you pull your hand back even before you consciously perceive that the stove is hot.

Re: undefined

#68

Post by StackOverlord »

RecycledEle wrote: post #32
RecycledEle wrote: post #30

Earlier quoted context omitted.

I agree. We need feedback loops on each joint that keep doing what they are doing until a higher feedback loop decides to change things. For example, as I enter this text, most of my fingers are holding the phone and do not need to be told to keep holding it. The low level controllers should also be able to respond to simple things. It's like when I get hurt, my hand yanks back before my brain realizes what has happe…

LLMs have terrible spatial awareness. This probably comes from ONLY being trained on text. I wonder if it would help a LLM running a robot to have a separate controller calculating it's position and what is around it, and feed that into the LLM constantly. It would be like when a video game has a radar display or map to let you know where you are in relation to other things.

LLMs have a firm grip on common sense. It's because it allows them to deal with the utterly unexpected they are deemed useful in robotics. Not to perform delicate movements, but stop doing so when police enters the room.

Re: undefined

#70

Post by NalNezumi »

lucidrains wrote: post #54
NalNezumi wrote: post #19

This seem like an cool upgrade from RT1, judging from the result. It seems to now also output the delta of the end-effector pose, which was previously handled by a different motion planner. It does seem like this work (and a lot of robot learning works) are still stuck on position/velocity control and not impedance control. Which is essentially output where to go, either closed-loop with a controller or open-loop wit…

thank you for sharing your thoughts! is there anyway I can get in touch with you, through email or some other means?

Sure thing! I'm a bit wary to share my email on public forum but you can DM me on Twitter! https://twitter.com/DanOgawaLil
Post reply on HN