Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hands-Free Coding: How I develop software using dictation and eye-tracking

joshwcomeau.com

Post reply on HN
Print view View original

81–90 of 141 posts

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#81

Post by melling »

cschneid wrote: post #80
melling wrote: post #65

Earlier quoted context omitted.

Not sure how old you are but I imagine over the next couple of decades we’ll eventually get more advanced computer interfaces. Gestures, voice, eye tracking, ... Being a software developer could be a job for those who physically can’t do other jobs. Subtle input gestures with Soli would be interesting. https://atap.google.com/soli/

Assuming physical injury (like RSI or trauma), yes, it's getting easier and easier. But there are many other ways you could be injured and unable to work. My straw-man injury when I bought the policy was: "I'm a new skier, which is famous for major head trauma". Similarly, cancer, which directly or via treatment, causes exhaustion is another example of disability that is not solved with "if only I could just input te…

Yes, that makes sense. However, the older I get the more I think society should provide some sort of safety net for the unlucky.

It’s inefficient for everyone to do this and many cannot.

The premiums could jump 50% in a decade, for example. I’ve known situations somewhat like yours where premiums went up a lot.

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#82

Post by ianhorn »

lunixbochs wrote: post #70
rntz wrote: post #66

Earlier quoted context omitted.

Talon actually can work with Dragon; it started out requiring Dragon but has since grown its own voice recognition. Anecdotally, a lot of folks on the talon slack seem to be refugees from Dragon-based solutions, and a common observation is that Talon's voice recognition is much lower-latency than Dragon. Unfortunately I can't confirm this as I haven't used Dragon myself. My impression of talon's speech recognition is…

> Talon actually can work with Dragon More specifically, "can work" means: if you run Talon on Windows or Mac alongside Dragon, and don't configure Talon to use its own engine, Talon will automatically use Dragon as the recognition engine.

> or Mac alongside Dragon

Unfortunately it looks like Dragon on Mac isn't a thing anymore, which is a shame because my main dev machine is a Mac. Or at least it used to be, before I needed to start dictating everything.

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#83

Post by rohith2506 »

I have suffered with thoracic outlet syndrome for almost 2 years. Countless physical therapy and deep tissue massages helped alleviate the pain but nothing seems to solve the root cause. I am 26 and the thought of not being able to use my hands to do something I have frightened the hell out of me.

That's when I stumbled upon John Sarno and his book "The mindbody prescription". I thought it's pseudoscience and I read it with very low expectations. And It's been almost four months since I have trouble using my computer.

In a nutshell, most of RSI injuries are not just physical problems and they are tightly coupled with your mind. The most important step is to acknowledge that subconscious thoughts related to pain and work along with it. Whenever I feel a tingling sensation, I yell at myself ( In my head ) that it's all in my mind and I have started to feel normal. Of course, this might not work for everyone but definitely worth a try.

And I also do lot of strength training and climbing which really helped strengthening my hands

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#84

Post by luckylion »

lunixbochs wrote: post #45
luckylion wrote: post #2

I love the eye tracking. Even without having any issues with my hands, I'd like to have quick mouse movements without having to take a hand off of the key board. Zooming in to have more precision is clever. It seems the current generation systems aren't working for Linux yet, but that gives me hope to have a way to work with it at some point in the future.

> It seems the current generation systems aren't working for Linux yet This isn’t correct. Talon supports eye tracking on all of Windows, Linux, and Mac with the same hardware.

Oh, that's great news. I thought Talon needed the drivers to work and was constrained in that regard. I will need to free up some time and try it out.

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#85

Post by rsa25519 »

rohith2506 wrote: post #83

I have suffered with thoracic outlet syndrome for almost 2 years. Countless physical therapy and deep tissue massages helped alleviate the pain but nothing seems to solve the root cause. I am 26 and the thought of not being able to use my hands to do something I have frightened the hell out of me. That's when I stumbled upon John Sarno and his book "The mindbody prescription". I thought it's pseudoscience and I read…

I'm not so sure about this. My understanding is that RSI is caused by inflammation, and ignoring it will only make it worse, to the point of needing surgery. Although it sounds like you've been dealing with this longer, so I'll give you the benefit of the doubt

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#86

Post by lunixbochs »

ianhorn wrote: post #82
lunixbochs wrote: post #70

Earlier quoted context omitted.

> Talon actually can work with Dragon More specifically, "can work" means: if you run Talon on Windows or Mac alongside Dragon, and don't configure Talon to use its own engine, Talon will automatically use Dragon as the recognition engine.

> or Mac alongside Dragon Unfortunately it looks like Dragon on Mac isn't a thing anymore, which is a shame because my main dev machine is a Mac. Or at least it used to be, before I needed to start dictating everything.

Talon can also use Dragon remotely from a Windows machine or VM, kind of like aenea but more tightly integrated (the host is driving instead of the guest).

And I do have some users voluntarily switching from Mac Dragon to the improved speech engine in the Talon beta. Mac Dragon has been kind of buggy for the last few years so you're not missing much.

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#87

Post by ianhorn »

cybwraith wrote: post #67
ianhorn wrote: post #62

I’ve been having similar issues lately, and after looking at talon, serenade, and caster, I ended up using caster [0]. The different programs all have significant differences in usability, and have clever ideas behind them, but unfortunately automatic speech recognition is still bad enough that the primary factor is which has a better ASR engine. Caster supports Dragon Naturally Speaking, which is expensive, but enou…

Do you know of any workflows like this using entirely open source software? EDIT: Seems like caster itself has instructions for an open source recognition engine to pair with it. Not sure how accurate it'll be but I'm going to give it a shot! I hate the idea of relying on closed source software to be able to continue in my profession. If this works I'm definitely going to be donating to the FOSS options

Yeah there is the Kaldi backend for caster, which I've tried on my Mac machine, since Dragon isn't a thing on Mac. Unfortunately it's not nearly as good :-(

I'd like to try to record my usage of Dragon that I could fine-tune my own model, but it's harder to get around to hobby coding projects like that now that coding is more of a pain in the ass.

The unfortunate reality in my case is that using Dragon is the least frustrating way to keep working. I don't think closed paid ASR models will stop being noticeably better better than three open ones until the state-of-the-art error rate is basically zero.

I do like that caster is open source where talon isn't, but preferences like that are pretty low priority when say your career is on the line.

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#88

Post by lunixbochs »

whimsicalism wrote: post #73
cybwraith wrote: post #67

Earlier quoted context omitted.

Do you know of any workflows like this using entirely open source software? EDIT: Seems like caster itself has instructions for an open source recognition engine to pair with it. Not sure how accurate it'll be but I'm going to give it a shot! I hate the idea of relying on closed source software to be able to continue in my profession. If this works I'm definitely going to be donating to the FOSS options

My understanding is that Talon is built off of wav2letter's inference framework for ASR.

I do not use the wav2letter@anywhere inference frontend - I trained the acoustic model using the facebook upstream code, but the decoder is almost entirely new, and on Windows I use Pytorch for inference.

Talon ships with a libw2l.so/dylib on Linux/Mac built from my open source repos.

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#89

Post by ianhorn »

lunixbochs wrote: post #86
ianhorn wrote: post #82

Earlier quoted context omitted.

> or Mac alongside Dragon Unfortunately it looks like Dragon on Mac isn't a thing anymore, which is a shame because my main dev machine is a Mac. Or at least it used to be, before I needed to start dictating everything.

Talon can also use Dragon remotely from a Windows machine or VM, kind of like aenea but more tightly integrated (the host is driving instead of the guest). And I do have some users voluntarily switching from Mac Dragon to the improved speech engine in the Talon beta. Mac Dragon has been kind of buggy for the last few years so you're not missing much.

Any chance you have pointers on how to set that up? You'd probably laugh/cry to see my setup right now, with my Windows desktop on the left monitors and my MacBook on the right monitors because I need both... purely because Dragon is only sold on Windows since this started been an issue for me. A more tightly coupled super-aenea sounds pretty fantastic.

Re: Hands-Free Coding: How I develop software using dictation and eye-tracking

#90

Post by OJFord »

T-hawk wrote: post #64
irjustin wrote: post #22

Earlier quoted context omitted.

The planck keyboard is a great keyboard, but possibly on par with the worst in terms of ergonomics. Ergodox, Diverge, Keyboard.io, really any split key keyboard does worlds for keeping hands straight and are fully programmable to very high degrees.

Am I the only one who just keeps their hands/wrists straight on a regular unsplit keyboard? I've always naturally done this and never had any problems with strain. The best way to describe it is that the home keys aren't ASDF, but more like WEFV with my forearm angled so it's straight from elbow to fingertip. This exaggerates the effect but demonstrates it clearly; the real home positions are more like intermediate p…

On my keyboard - which is.. I don't want to say 'full size', because maybe it's '80%' or whatever, but it's a regular external keyboard with numpad - f & j are centre to centre 60mm apart.

If I outstretch my arms, straight, my index fingers are pretty much in line with my shoulders.

So no, when I use my regular unsplit keyboard my hands/wrists are not straight, because my shoulders are not 60mm apart..? Am I missing something?

Post reply on HN