11–20 of 63 posts
I can’t help you with actually converting speech to code but it occurs to me this would be a benefit to everyone. Speaking the words that are represented by the code we write would require a much deeper understanding of what we are doing and why. Food for thought for sure. Good luck.
Not coding - but openai's API in beta has a speech to bash function https://openai.com/blog/openai-api/
I haven't seen any demos actually using speech recognition as the input, but I'm excited to try that as soon as I'm off the waitlist. One of the caveats I have in mind is that speech recognition language models are going to be worse than GPT3, so the best option might be getting GPT3 to do the language decoding in-domain _and_ do the task. (The other caveat I have is it seems pretty slow for interactive speech use, but I have some ideas on how the knowledge could be cached in a sense over time.)
I develop kaldi-active-grammar [0]. The Kaldi engine is state of the art for command and control. Although I don't have the data and resources for training a model like Microsoft/Nuance/Google, being an open rather than closed system allows me to train models that are far more personalized than the large commercial/generic ones you are used to. For example, see the video of me using it [1], where I can speak in a relaxed manner without having to over enunciate and strain my voice.
Gathering the data for such training does take some time, but the results can be huge [2]. Performing the actual training is currently complicated; I am working on making it portable and more turnkey, but it's not ready yet. However, I am running test training for some people. Contact me if you want me to use you as a guinea pig.
[0] https://github.com/daanzu/kaldi-active-grammar
[1] https://youtu.be/Qk1mGbIJx3s
[2] https://github.com/daanzu/kaldi-active-grammar/blob/master/d...
Also, talk to an ergonomics person about it, and it sounds like notebooks are out at this point unless you have an external keyboard, mouse and monitor.
Not coding - but openai's API in beta has a speech to bash function https://openai.com/blog/openai-api/
I can’t help you with actually converting speech to code but it occurs to me this would be a benefit to everyone. Speaking the words that are represented by the code we write would require a much deeper understanding of what we are doing and why. Food for thought for sure. Good luck.
There's been some pretty cool work in this area recently: https://www.youtube.com/watch?v=fZSFNUT6iY8
In my experience, I've found any services claiming to do deep learning produced far worse results than what we could get with simple approaches. That is, when faced with non-grammatical sentences (or rather, sentences with a different grammar than English's). Of course that's because models are not typically trained with this use-case in mind! But the fact that you need a huge load of data to even slightly alter the expected inputs of the system, to me, was a deal breaker.
For the specific case of programming with voice, Silvius comes to mind. It's built and used by a developer with this same problem. It's a bit wonky having to spell words sometimes with alpha-beta-gamma speech, and it won't work without some customization, but on the other hand it's completely free and open source: https://github.com/dwks/us
I've been working on a similar use case at work (going from discoursive speech to cli-like commands, using a semi-rigid language), and I didn't find any off-the-shelf purely ML-based solution that would work for us. In my experience, I've found any services claiming to do deep learning produced far worse results than what we could get with simple approaches. That is, when faced with non-grammatical sentences (or rath…
That said, Silvius is more of demo than a product, the IMO best voice programming options right now are (in alphabetical order):
- Caster/dragonfly (fully open-source if you use daanzu's Kaldi engine, which is way better than Silvius afaik, I think even the creator of silvius uses dragonfly with dragon instead of using silvius)
- Serenade (fully commercial, I haven't looked at it much recently but biggest caveats afaik are accuracy, the fact speech recognition is web based, and it's restricted to specific languages and IDEs while caster/talon are for full system control and not just programming)
- Talon (my project, semi-commercial as I work on it full time and draw income from it but aim to give all necessary features away for free, some benefits include a fully offline and open-source speech recognition engine, and I have other bonuses like eye tracking and noise recognition)