The only way to prevent robots from being jailbroken and set to rob banks is to move GPUs to private SOTA secure GPU clouds
Gemini Robotics On-Device brings AI to local robotic devices
61–70 of 92 posts
Re: Gemini Robotics On-Device brings AI to local robotic devices
#62The MuJoCo link actually points to https://github.com/google-deepmind/aloha_sim
mujoco_menagerie has Mujoco MJCF XML models of various robots.
google-deepmind/mujoco_menagerie: https://github.com/google-deepmind/mujoco_menagerie
mujoco_menagerie/aloha: https://github.com/google-deepmind/mujoco_menagerie/tree/mai...
Re: Gemini Robotics On-Device brings AI to local robotic devices
#63I've spent the last few months looking into VLAs and I'm convinced that they're gonna be a big deal, ie they very well might be the "chatgpt moment for robotics" that everyone's been anticipating. Multimodal LLMs already have a ton of built-in understanding of images and text, so VLAs are just regular MMLLMs that are fine-tuned to output a specific sequence of instructions that can be fed to a robot. OpenVLA, which c…
I don't think transformers will be viable for self driving cars until they can both: 1) Properly recognize what they are seeing without having to lean so hard on their training data. Go photoshop a picture of a cat and give it a 5th leg coming out of it's stomach. No LLM will be able to properly count the cat's legs (they will keep saying 4 legs no matter how many times you insist they recount). 2.) Be extremely fast…
The current gen VLA architectures include some tricks (like compressed action tokenization and diffusion decoding) to reach action frequencies between 50-200hz. I think they’re _more_ efficient this way than regular LLMs trying to do everything thru text.
Re: Gemini Robotics On-Device brings AI to local robotic devices
#64Nice. I work with some students younger than 13, so most cloud and llms are quite tricky to work with, local only models like vertex are nice for this use case. I will try this as a replacement for chatgpt as Computer Vision in robotics like Lego Mindstorm
Re: Gemini Robotics On-Device brings AI to local robotic devices
#65Earlier quoted context omitted.
Actually very close to one I'd say. It's a "visual language action" VLA model "built on the foundations of Gemini 2.0". As Gemini 2.0 has native language, audio and video support, I suspect it has been adapted to include native "action" data too, perhaps only on output fine-tuning rather than input/output at training stage (given its Gemini 2.0 foundation). Natively multimodal LLM's are basically brains.
OpenVLA is basically a slightly modified, fine-tuned llama2. I found the launch/intro talk by lead author to be quite accessible: https://www.youtube.com/watch?v=-0s0v3q7mBk
A more modern one, smolVLA is similar and uses a VLM but skips a few layers and uses an action adapter for outputs. Both are from HF and run on LeRobot.
https://arxiv.org/abs/2506.01844
Explanation by PhosphoAI: https://www.youtube.com/watch?v=00A6j02v450
Re: Gemini Robotics On-Device brings AI to local robotic devices
#66These are going to be war machines, make absolutely no mistake about it. On-device autonomy is the perfect foil to escape centralized authority and accountability. There’s no human behind the drone to charge for war crimes. It’s what they’ve always dreamed of. Who’s going to stop them? Who’s going to say no? The military contracts are too big to say no to, and they might not have a choice. The elimination of toil wil…
MIT spinoff Google-owned Boston Dynamics pledged not to militarize their robots. Which is very hard to believe given they're backed by DARPA, the DoD/Military investment arm.
This pledge would last five seconds in an actual conflict, if it makes it even that far.
Re: Gemini Robotics On-Device brings AI to local robotic devices
#67Earlier quoted context omitted.
To an extent, yes. For example, an industrial robot arm with 6 motors achieves much higher reliability than a consumer roomba with 3 motors. They do this with more metal parts, more precision machining, much more generous design tolerances, and suchlike. Which they can afford by charging 100x as much per unit.
Also factory robots arms are probably operating in highly sterile, dry environments? How would working in a muddy / dusty / wet environment change this?
When designing hardware, you usually define what the expected operating environments are. Some typical environmental considerations are the min/max temperature, debris ingress, shock & vibration. If you know your product is going to operate in an area where material is likely to enter the product, then you can either try to keep that material out (sealing the product up), or make sure that dust entering the product won't cause failures (i.e. electrical shorts won't happen on a board by covering exposed areas with glue or making sure a mechanism can crush/clear particles). It's not necessarily more complexity in the product to navigate these constrains, but it is another thing to consider in the design.
For example, if you're making a phone that is going to be sold around the world, then you're going to worry about arctic/equator temps (will some of your components melt or ICs fail), salty sea air (will the product begin to corrode for people living by a beach), or fast moving elevators (will the speakers pop from a sudden change in pressure).
You can check out this manufacturers robot arms as some examples of existing products. They list some data sheets for their robot arms, including some arms that are IPxx rated. I don't think looking at robot arms is a 1to1 comparison for what you could expect from a humanoid robot since the considerations in the design process are going to be different.
website is kuka dot com/en-at/products/robotics-systems/industrial-robots/kr-agilus
Re: Gemini Robotics On-Device brings AI to local robotic devices
#68Earlier quoted context omitted.
It does either get very exciting or very spooky thinking of the possibilities in the near future. I had always assumed that such a robot would be very specific (like a cleaning robot) but it does seem like by the time they are ready they will be very generalizable. I know they would require quite a few sensors and motors, but compared to self-driving cars their liability would be less and they would use far less mate…
The exciting part comes when two robots are able to do repairs on each other.
But this still has a massive cost. Replacing or repairing an actuator isn't cheap, in material and in time of unavailability.
Re: Gemini Robotics On-Device brings AI to local robotic devices
#69Earlier quoted context omitted.
To an extent, yes. For example, an industrial robot arm with 6 motors achieves much higher reliability than a consumer roomba with 3 motors. They do this with more metal parts, more precision machining, much more generous design tolerances, and suchlike. Which they can afford by charging 100x as much per unit.
Also factory robots arms are probably operating in highly sterile, dry environments? How would working in a muddy / dusty / wet environment change this?
Search for CNC videos, those machines work in oily, soapy, dusty and full of metal shavings environments and do fine.
Re: Gemini Robotics On-Device brings AI to local robotic devices
#70Earlier quoted context omitted.
Also factory robots arms are probably operating in highly sterile, dry environments? How would working in a muddy / dusty / wet environment change this?
Search for CNC videos, those machines work in oily, soapy, dusty and full of metal shavings environments and do fine.
It's still a fairly controlled environment with splash guards, liquid based dusts suppression and or dust collection, even then my friend has a factory and the CNC is a reliable machine but things screw up.
If the dust collection was disabled, the workshop and the machine would be caked in debris.
It doesn't move, it doesn't fall over or have anything falling on top of it either (like a robot could).