Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hands-Free Coding: How I develop software using dictation and eye-tracking

joshwcomeau.com

Post reply on HN
Print view View original

61–70 of 141 posts

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#61

Post by tmacwilliam »

CoffeePython wrote: post #46
melling wrote: post #38

The author links to this other project, which I’ve never heard of: https://serenade.ai/ He also references Tavis Rudd’s viral voice coding video, which is already 7 years old. https://www.youtube.com/watch?v=8SkdfdXWYaI The future is arriving slower than i would have guessed. I thought we’d be developing on the iPad using voice, gestures, and eye tracking by now. Can I a least get “build and run” with my voice soon?

I'm very surprised none of the videos on the serenade landing page have audio. I would love to see/hear what it's like. I'd suspect adding audio to those videos would be a win on conversions.

Thanks for the suggestion! We recently revamped our homepage, and we're in the process of recording some new videos. Will keep that in mind!

In the meantime, here's a video that a developer using Serenade recently made about their first experience! https://www.youtube.com/watch?v=Pc-EbY1fRWk

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#62

Post by ianhorn »

I’ve been having similar issues lately, and after looking at talon, serenade, and caster, I ended up using caster [0]. The different programs all have significant differences in usability, and have clever ideas behind them, but unfortunately automatic speech recognition is still bad enough that the primary factor is which has a better ASR engine. Caster supports Dragon Naturally Speaking, which is expensive, but enough better to make it worthwhile.

There are moments where I think this is going to be the future of programming, since code as text is only the easiest for as long as typing is the easiest way to record things unambiguously.

But for the most part it’s still pretty frustrating. If ASR systems can get the sentence error rate down by an order of magnitude or two, I sincerely think this will take off not just for accessibility, but for normal use.

Until then, it’s a PITA that is saving my career nonetheless.

[0] https://caster.readthedocs.io/en/latest/

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#63

Post by vectorphresh »

baxtr wrote: post #29

Reminds me of the Figma GPT-3 Plugin https://mobile.twitter.com/edleonklinger/status/128425141917...

I wonder if the author has considered integrating his voice interface with an AI resource like GPT-3. I could see future development being done in this fashion.

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#64

Post by T-hawk »

irjustin wrote: post #22
867_5309_Jenny wrote: post #10

Earlier quoted context omitted.

Switching to a planck keyboard would let you customize all the positions of buttons for less strain.

The planck keyboard is a great keyboard, but possibly on par with the worst in terms of ergonomics. Ergodox, Diverge, Keyboard.io, really any split key keyboard does worlds for keeping hands straight and are fully programmable to very high degrees.

Am I the only one who just keeps their hands/wrists straight on a regular unsplit keyboard? I've always naturally done this and never had any problems with strain.

The best way to describe it is that the home keys aren't ASDF, but more like WEFV with my forearm angled so it's straight from elbow to fingertip. This exaggerates the effect but demonstrates it clearly; the real home positions are more like intermediate points between keys.

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#65

Post by melling »

cschneid wrote: post #47
Symbiote wrote: post #24

When I started my undergraduate degree, there was a PhD student everyone knew -- he looked like he ran the Linux User Group, which he did, he coded his own window manager, wrote games in Haskell etc. At some point, he got some hand injury that meant he couldn't type normally for months. He would only have been 21 or 22. This scared me, so since then I've followed the standard advice on avoiding computer-related injur…

A few years back I started buying disability insurance, in addition to paying more attention to my ergonomics. My greatest economic asset is the years left in my career. For ~$1500/year I insure that. It's not cheap, but it would pay out until I was 65 at some reasonable percentage of my normal salary in the case I was severely injured, or otherwise unable to work. Anywhere from an ergonomics caused hand/arm injury,…

Not sure how old you are but I imagine over the next couple of decades we’ll eventually get more advanced computer interfaces.

Gestures, voice, eye tracking, ...

Being a software developer could be a job for those who physically can’t do other jobs.

Subtle input gestures with Soli would be interesting.

https://atap.google.com/soli/

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#66

Post by rntz »

ianhorn wrote: post #62

I’ve been having similar issues lately, and after looking at talon, serenade, and caster, I ended up using caster [0]. The different programs all have significant differences in usability, and have clever ideas behind them, but unfortunately automatic speech recognition is still bad enough that the primary factor is which has a better ASR engine. Caster supports Dragon Naturally Speaking, which is expensive, but enou…

Talon actually can work with Dragon; it started out requiring Dragon but has since grown its own voice recognition. Anecdotally, a lot of folks on the talon slack seem to be refugees from Dragon-based solutions, and a common observation is that Talon's voice recognition is much lower-latency than Dragon. Unfortunately I can't confirm this as I haven't used Dragon myself.

My impression of talon's speech recognition is that it's good enough for voice coding, but could be improved when it comes to dictating English prose. That said, there are promising avenues of improvement. If you pay for the Talon beta, there's a more advanced voice recognition engine available that's much better at English, and you can also hook into Chrome's voice recognition for dictation.

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#67

Post by cybwraith »

ianhorn wrote: post #62

I’ve been having similar issues lately, and after looking at talon, serenade, and caster, I ended up using caster [0]. The different programs all have significant differences in usability, and have clever ideas behind them, but unfortunately automatic speech recognition is still bad enough that the primary factor is which has a better ASR engine. Caster supports Dragon Naturally Speaking, which is expensive, but enou…

Do you know of any workflows like this using entirely open source software?

EDIT: Seems like caster itself has instructions for an open source recognition engine to pair with it. Not sure how accurate it'll be but I'm going to give it a shot!

I hate the idea of relying on closed source software to be able to continue in my profession. If this works I'm definitely going to be donating to the FOSS options

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#68

Post by azinman2 »

For RSI folks, my experience is that RSI can “move” to other parts of the body pretty easily, including eyes and vocal chords. Somehow doing “work” with a part of the body can strain it, even when talking all day long socially wouldn’t.

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#69

Post by lunixbochs »

ornornor wrote: post #59
lunixbochs wrote: post #45

Earlier quoted context omitted.

> It seems the current generation systems aren't working for Linux yet This isn’t correct. Talon supports eye tracking on all of Windows, Linux, and Mac with the same hardware.

I think parent is talking about the Tobii. I was also very very excited about it (I can’t find a mouse or trackball that won’t give me hand or shoulder or wrist pain) but then I saw it only works on windows... and nothing else. No Linux support. So that’s a no go. If someone knows of such a device for Linux, I’d love to hear about it!

Let me be more specific. The Tobii 4c, PCEye Mini, and Tobii 5* work approximately the same on Linux, Mac, and Windows with Talon out of the box.

* The Tobii 5 does work, but is a work in progress and has some caveats until I do more work on it.

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#70

Post by lunixbochs »

rntz wrote: post #66
ianhorn wrote: post #62

I’ve been having similar issues lately, and after looking at talon, serenade, and caster, I ended up using caster [0]. The different programs all have significant differences in usability, and have clever ideas behind them, but unfortunately automatic speech recognition is still bad enough that the primary factor is which has a better ASR engine. Caster supports Dragon Naturally Speaking, which is expensive, but enou…

Talon actually can work with Dragon; it started out requiring Dragon but has since grown its own voice recognition. Anecdotally, a lot of folks on the talon slack seem to be refugees from Dragon-based solutions, and a common observation is that Talon's voice recognition is much lower-latency than Dragon. Unfortunately I can't confirm this as I haven't used Dragon myself. My impression of talon's speech recognition is…

> Talon actually can work with Dragon

More specifically, "can work" means: if you run Talon on Windows or Mac alongside Dragon, and don't configure Talon to use its own engine, Talon will automatically use Dragon as the recognition engine.

Post reply on HN