Ah nice I've tinkered a bit with robotics few years ago, e.g. https://twitter-archive.benetou.fr/utopiah/status/1760260544... and it's pretty empowering... but also showcases how amazing the human body is. Our dexterity, ability to sense, etc. Reminds me of https://rodneybrooks.com/why-todays-humanoids-wont-learn-dex... which is basically a stark warning against the hype.
Cool video! What made you stop if I can ask? And yeah I feel you re humanoid. I worked on the Rubik's cube project at OpenAI, which used a humanoid hand, and it was insanely painful and hard. Also fun anecdote: it was completely impossible to teleop the shadow hand. We had a data glove to capture hand movements but as soon as contact / haptics come in, you're lost. We could never even get a single rotation on the Rub…
I would have liked to explore a bit for but doing innovation readiness on technologies related to XR I basically had to move on to the next project. This though happened mostly because, as pointed out by the essay by Brooks, dexterity is hard, on both ends. Namely yes one can get a basic robot "arm" for cheap... but doing a robotic arm with a hand is something else, one with fine motor control is ... well I'm not an expert in the field but basically it doesn't exist yet IMHO. Sure we have grippers but that has basically nothing to do with a human hand. It's amazing how much flexibility we have at our disposal in such a compact and efficient form, sensors for touch obviously but also heat, proprioception of course, part that are smooth and flexibly while other are hard. The range of craftmanship we can do is... mindblowing. If you don't believe me just look at a basic magician, not even a good one (like me, I confess), doing sleight of hand, it's just amazing.
So that was on the robotic part, discovering what it could do, amazing, but more importantly for my work what it could NOT do.
A seemingly simple project was how to remove a 3D print from our 3D printer in order to free it up and move to the next job. This sounds trivial ... until you try to actually do it. I won't get into details but we didn't manage. It's of course feasible in the ideal scenario, e.g. successful print that is mostly rigid with attachment and support to the plate that requires just the right amount of force. It can be done. Now doing that in a realistic set of scenarii that a 3D printing house would do... well maybe it's feasible but I professionally didn't know (and still don't know) how to do with a realistic set of constraints (time wise, economically speaking too).
Moving on then to the other end, or hand (sorry for the pun) tracking in VR is good, honestly. It's quite fun for games... then trying to do so in a professional scenario then 1mm difference or occlusion for .1s is not acceptable anymore.
TL;DR (sorry I have to run and it might be longer than what you even asked for!) : the concept itself is obviously good, especially from a programmer standpoint. We are expert at automating, in fact I'd argue that's the 1 thing we excel at. The implementation though, in real life, is much harder that we naively consider, even with a LOT more computing power.
TL;DR (short): quick wins, yes, harder wins... not intractactable but at least beyond my own ability.