Ask HN: Is there any work being done in speech-to-code with deep learning? • Hacker News Board

Re: Ask HN: Is there any work being done in speech-to-code with deep learning?

#41

Post by mkl » Sat, Jun 13, 2020, 10:35 AM UTC

I use dictation a bit for prose, but my voice wouldn't be able to handle more than a couple of hours a day of that.

Can you use a touch screen or mouse? I went ~13 years without using a keyboard, and typed with mice (some customised), trackballs, and touch screens, mostly using predictive typing software I wrote. In that time I did a lot of programming, including a whole applied maths PhD.

One of the best mouse setups I came up with a variety of versions of was moving the cursor with one hand, and clicking with the other. Holding the mouse still to click the button accurately is a surprisingly problematic movement. I made a button-less mouse with just a flat top to rest the side of my hand on, with a bit sticking up to grip. Standalone USB numeric keypads can be remapped to mouse clicks and common keys.

Touch screens can also be very good, if set up right, as all the movement can come from the big muscles and joints of your upper arm and shoulder, and your fingers and wrist don't need to do much. The screen needs to be positioned well, not out in front of you, but down close and angled in a comfortable position to hold your arm for long periods.

Re: Ask HN: Is there any work being done in speech-to-code with deep learning?

#42

Post by GordonS » Sat, Jun 13, 2020, 10:46 AM UTC

daanzu wrote: post #14

Windows Speech Recognition is far from the best, so perhaps your trouble could be partly caused by how you had to speak in order to be understood, rather than the command style? I used to use WSR to code by voice, and it was far more laborious than my current setup. I develop kaldi-active-grammar [0]. The Kaldi engine is state of the art for command and control. Although I don't have the data and resources for traini…

> Performing the actual training is currently complicated; I am working on making it portable and more turnkey, but it's not ready yet

I'm eagerly awaiting this! If I wanted to try to get something working now, I'd need to invest a lot of time - being able to get started quickly would be amazing.

Re: Ask HN: Is there any work being done in speech-to-code with deep learning?

#43

Post by raidicy » Sat, Jun 13, 2020, 11:03 AM UTC

tluyben2 wrote: post #40

Probably not good for your case, but end of the summer we are going in beta launch for our product which is a visual + speech controlled programming language. It's very niche as it's a new language and IDE, from scratch, but so far it's been fun working on it.

i'd be interested in seeing it. can i follow you somewhere

Re: Ask HN: Is there any work being done in speech-to-code with deep learning?

#44

Post by fareesh » Sat, Jun 13, 2020, 11:19 AM UTC

I remember watching a video of this guy from maybe 10-12 years ago who used IBM Dragon natural speak into some kind of vim commands and shorthand language which enabled him to write code faster than he could using the keyboard. It was a fascinating demo, but the efficiency seemed to come from his short language that he developed for his workflow like "bop chop hop" etc.

Re: Ask HN: Is there any work being done in speech-to-code with deep learning?

#45

Post by phrotoma » Sat, Jun 13, 2020, 11:27 AM UTC

fareesh wrote: post #44

I remember watching a video of this guy from maybe 10-12 years ago who used IBM Dragon natural speak into some kind of vim commands and shorthand language which enabled him to write code faster than he could using the keyboard. It was a fascinating demo, but the efficiency seemed to come from his short language that he developed for his workflow like "bop chop hop" etc.

Probably this clip of Tavis - https://youtu.be/8SkdfdXWYaI?t=530?

Re: Ask HN: Is there any work being done in speech-to-code with deep learning?

#46

Post by daanzu » Sat, Jun 13, 2020, 11:42 AM UTC

mkl wrote: post #41

I use dictation a bit for prose, but my voice wouldn't be able to handle more than a couple of hours a day of that. Can you use a touch screen or mouse? I went ~13 years without using a keyboard, and typed with mice (some customised), trackballs, and touch screens, mostly using predictive typing software I wrote. In that time I did a lot of programming, including a whole applied maths PhD. One of the best mouse setup…

For me at least, dictation is actually the more straining mode of speech recognition, as compared to using my command grammars. With dictation, you might say anything, so the computer is given wide leeway in what to recognize, and so you must speak as clearly as possible. With commands (especially a nice simple command grammar), however, what you can say is greatly restricted, which allows you the freedom to speak indistinctly and still be understood by the computer. This can even be magnified by personalized training of the speech recognition model.

When using commands at my computer, I frequently find myself muttering and grunting things that even I think to myself (that is utterly un-understandable), yet the computer understands just fine. Dictating for prolonged periods can be tiring for me, but I can happily code by voice commands all night.

More info in my other comments here: https://news.ycombinator.com/item?id=23507363 https://news.ycombinator.com/item?id=23507829

Re: Ask HN: Is there any work being done in speech-to-code with deep learning?

#47

Post by salt-licker » Sat, Jun 13, 2020, 12:01 PM UTC

apeddle wrote: post #5

I came across Serenade ( https://serenade.ai/ ) recently. It's still beta but I was very impressed. In the past I've used vocola, and a few other open-source options. Serenade felt much more natural and powerful. The founders are also super hands-on and genuinely seem to care about the problem.

Yes, one of the founders is a coder who developed RSI and couldn’t find a good tool. In a form of UX bootstrapping, I’m pretty sure the product is being used to build itself

Re: Ask HN: Is there any work being done in speech-to-code with deep learning?

#48

Post by xchaotic » Sat, Jun 13, 2020, 12:21 PM UTC

Not sure if off topic but it wasn’t the RSI that put the nail in the coffin for my programming. It was spine injury and I had to have a surgery. There’s lots of jobs around programming that don’t require as much typing and even when you do, it’s easier to dictate email than code. Basically you got RSI from coding and generally spending too much time with the keyboard. Maybe at least consider alternatives where you are not spending lots of screen time again.