I've been wanting something like Talon for years. Easily customized voice recognition for commands (you hear me say 'blah', you type/start a program/click a button labelled 'blah') is something I've not had since I was very young and tried to trick out LCARS x32 using only Windows XPs speech recognition software using the crappy stick mics that lived on top of every single beige CRT monitor at school.
Hands-Free Coding: How I develop software using dictation and eye-tracking
41–50 of 141 posts
Re: Hands-Free Coding: How I develop software using dictation and eye-tracking
#42Earlier quoted context omitted.
In addition to a pointer replacement, there is an idea of using eye tracking on a virtual keyboard with a swype type entry.
I dunno... "swyping" your eyes around a virtual keyboard seems like it'd cause a lot of strain on your eyes.
There are alternative virtual keyboard styles (typically made for use with a mouse and impaired motor control) which would work better. Fewer "fine" movements.
Combined with predictive typing, and you could go pretty far.
Re: Hands-Free Coding: How I develop software using dictation and eye-tracking
#43Not sure if this helps but I had a similar issue with my joints (arm, shoulder, wrist) where I would scream in pain when lifting a small book or anything above 3 lbs. The issue was caused by my immune system acting up which would cause my tendons to become inflamed. After a year of misdiagnosis, Doctors found some heavy medication (methotrexate balanced with plaquenil) to help regulate it. One of the medicines side e…
I was misdiagnosed inflamed tendons and it turned out to be something else, hah
HN Reader take this message: medicine can be a to and fro, go back if you're not convinced and get a second opinion if you think somethings still wrong :)
Re: Hands-Free Coding: How I develop software using dictation and eye-tracking
#44Not sure if this helps but I had a similar issue with my joints (arm, shoulder, wrist) where I would scream in pain when lifting a small book or anything above 3 lbs. The issue was caused by my immune system acting up which would cause my tendons to become inflamed. After a year of misdiagnosis, Doctors found some heavy medication (methotrexate balanced with plaquenil) to help regulate it. One of the medicines side e…
Are you me?
Re: Hands-Free Coding: How I develop software using dictation and eye-tracking
#45I love the eye tracking. Even without having any issues with my hands, I'd like to have quick mouse movements without having to take a hand off of the key board. Zooming in to have more precision is clever. It seems the current generation systems aren't working for Linux yet, but that gives me hope to have a way to work with it at some point in the future.
> It seems the current generation systems aren't working for Linux yet
This isn’t correct. Talon supports eye tracking on all of Windows, Linux, and Mac with the same hardware.
Re: Hands-Free Coding: How I develop software using dictation and eye-tracking
#46The author links to this other project, which I’ve never heard of: https://serenade.ai/ He also references Tavis Rudd’s viral voice coding video, which is already 7 years old. https://www.youtube.com/watch?v=8SkdfdXWYaI The future is arriving slower than i would have guessed. I thought we’d be developing on the iPad using voice, gestures, and eye tracking by now. Can I a least get “build and run” with my voice soon?
I'm very surprised none of the videos on the serenade landing page have audio.
I would love to see/hear what it's like. I'd suspect adding audio to those videos would be a win on conversions.
Re: Hands-Free Coding: How I develop software using dictation and eye-tracking
#47When I started my undergraduate degree, there was a PhD student everyone knew -- he looked like he ran the Linux User Group, which he did, he coded his own window manager, wrote games in Haskell etc. At some point, he got some hand injury that meant he couldn't type normally for months. He would only have been 21 or 22. This scared me, so since then I've followed the standard advice on avoiding computer-related injur…
A few years back I started buying disability insurance, in addition to paying more attention to my ergonomics.
My greatest economic asset is the years left in my career. For ~$1500/year I insure that. It's not cheap, but it would pay out until I was 65 at some reasonable percentage of my normal salary in the case I was severely injured, or otherwise unable to work. Anywhere from an ergonomics caused hand/arm injury, a bike-riding caused head injury, a wood-shop caused eye damage, etc.
Consider grabbing it while you're young and working in a lucrative career.
Re: Hands-Free Coding: How I develop software using dictation and eye-tracking
#48The demo is incredible and heartening. I did not know the state of the art had improved so much. Thank you for sharing this, Josh.
Re: Hands-Free Coding: How I develop software using dictation and eye-tracking
#49When I started my undergraduate degree, there was a PhD student everyone knew -- he looked like he ran the Linux User Group, which he did, he coded his own window manager, wrote games in Haskell etc. At some point, he got some hand injury that meant he couldn't type normally for months. He would only have been 21 or 22. This scared me, so since then I've followed the standard advice on avoiding computer-related injur…
I have a similar story. When I was in college I got a repetitive stress injury from practicing piano too much. I ended up becoming a programmer but I have followed ergonomic practices religiously in the 40 years since then. My kids make affectionate fun of my perfect posture. But I have worked with the computer keyboard for going on five decades and have never come close to suffering a repetitive stress injury.
Re: Hands-Free Coding: How I develop software using dictation and eye-tracking
#50I don't know why, but I've often thought about the situation the Josh finds himself in. IE - having to find a different way of building software if I couldn't use a keyboard and mouse due to medical condition.
The solution I always imagine is paying someone or having my employer pay someone to strong style pair program with me. Perhaps a student, junior developer or even someone unfamiliar with software development entirely.
For those unfamiliar with strong style, this rule sums it up: "For an idea to go from your head into the computer it MUST go through someone else's hands". Like the standard driver / navigator pair programming technique but with the navigator never touching the keyboard.
In the case of someone completely unfamiliar with software development I imagine that there would initially be a dramatic high / low skill gradient between us, with the person essentially transcribing. However given the intensity of the practise I think this gradient would level out quite quickly.