Live data from Hacker News

Ask HN: Is there any work being done in speech-to-code with deep learning?

news.ycombinator.com

61–63 of 63 posts

Re: Ask HN: Is there any work being done in speech-to-code with deep learning?

#61
post #58

Earlier quoted context omitted.

It looks like Kaldi can use different backends, which I imagine have very different performance characteristics. Can you rank them from best to worst, with relative distances?

Just to be clear, the Dragonfly speech recognition command and control framework has multiple "backends" (speech recognition engines), including my Kaldi one. Probably the most used one currently is the Dragon Naturally Speaking backend. The Kaldi engine, being developed primarily for research in speech recognition, can support a huge variety of "models". I think the consensus general best for most use cases (particu…

Thank you, I think I understand partially, but not fully, as I'm not very well versed in speech recognition software.

Basically, my question (and I assume many other users') is "I run , what are my options and how good will my recognition be with each?". Your answer above helps, but it doesn't entirely satisfy me, as I'm not sure if a model is the recognition engine, or if the engine uses the model, or how I can use it, etc.

Re: Ask HN: Is there any work being done in speech-to-code with deep learning?

#62
Though unfortunately I cannot provide you with the conversational solution you are looking for, I believe there are some steps you can take/solutions that are currently available and that could help make your voice programming experience less exhausting, so it might be worth it if you gave them a try

1) try to minimize the amount you have to speak by leveraging auto completion as much as possible. For me TabNine [1] has been great help in that regard

2) try to use snippets as much as possible to reduce boilerplate code and because you can simply tab through the various fields. For me it has been great help that with sublime it is possible [2] without installing anything to have all of my snippets inside dragonfly grammars or even generate them dynamically [10] providing for much-needed structural control over what you write. I know this is more primitive (at least for the time being, there are ideas to improve it) than what you are asking for but for me it has been enough to make C++ enjoyable again! unfortunately my pull request to integrate this into Caster [3] has fallen behind but all of the basic functionality along with various additional utilities is there if you want to give it a try. Just be aware of these little bugger [4] that applies here as well!

3) not directly related to code generation but if you find yourself spending a lot of time and vocal effort for navigation consider either adding eye tracking to the mix or utilizing one of the at least three project that provide syntactical navigation capabilities. As author and more importantly as a user of PythonVoiceCodingPlugin [5], I have seen quite a bit of difference since I got it up to speed, because a) even though it is command driven ,command sound natural and smooth b) though they can get longer ,in practice utterances are usually 3 to 5(maybe 6) words , which makes them long enough so that you do not to speak abruptly but short enough that you do not have to hurry to speak them before you run out of breath c) and yeah I personally need less commands compared to using only keyboard shortcuts so less load for your voice! The other two project in this area I am aware of are Serenade [6] and VoiceCodeIdea [7] so see if something fits your use case!

4) use noise input where you can to reduce voice strain. Talon [8][9] is by far the way to go in this field but you might be able to get inferior but decent results with other engines as well. For instance, DNS 15 Home can recognize some 30+ letter like "sounds" like "fffp,pppf, tttf,shhh,ssss/'s,shhp,pppt,xxxx,tttp,kkkp" , you just have to make sure that you use 4 or more letters in your grammar (so for instance ffp will not work). recognition accuracy is going to degrade if you overloaded too much but it is still good enough to simplify a lot of common tasks.

5) give it a try with a different engine, I was not really that much satisfiedwith WSR either

6) see if any of the advise from [11] helps and seek out professional help!

I realize that my post diverges from what you originally asked for but I feel the points raised here might help you lessen the impact of voice strain for the time being until more robust solutions like the gpt3 mentioned in one of the comments above are up and running. My apologies if this is completely off topic!

[1] https://www.tabnine.com/ [2] https://github.com/mpourmpoulis/CasterSublimeSnippetInterfac... [3] https://github.com/dictation-toolbox/Caster [4] https://github.com/mpourmpoulis/PythonVoiceCodingPlugin/issu... [5] https://packagecontrol.io/packages/PythonVoiceCodingPlugin [6] https://serenade.ai/ [7] https://plugins.jetbrains.com/plugin/10504-voice-code-idea [8] https://talonvoice.com/ [9] https://noise.talonvoice.com/ [10] https://github.com/mpourmpoulis/CasterSublimeSnippetInterfac... [11] https://dictation-toolbox.github.io/dictation-toolbox.org/vo...

Re: Ask HN: Is there any work being done in speech-to-code with deep learning?

#63
I have similar issues. I use a combination of kinesis advantage, penguin mouse, and dragonfly/DNS. Having a good microphone does make a difference, as does retraining/tweaking command vocab. The biggest thing overall is the ergonomics of desk work - I take a break every 15 minutes (or try to) by setting timers.
Post reply on HN