Looks similar to the new FUTO keyboard: https://voiceinput.futo.org/
This looks great! I've been wanting to drop the Swipe keyboard ever since I saw sneaky ads on it (like me typing "Google Maps" and getting "Bing Maps" as a "suggestion").
Transcribro: On-device Accurate Speech-to-text
51–60 of 66 posts
Re: Transcribro: On-device Accurate Speech-to-text
#52Earlier quoted context omitted.
This is fascinating because if your hint in another comment indicates you worked on this at Google, it's entirely possible I have this all wrong because I'm missing the actual ML part - I wrote the client encoder & server decoder for Opus and the client-side UI for the SODA launch, and I'm honestly really surprised to hear Google has different stuff. The client-side code loop AGSA used is 100% replicated, in my exper…
Something to keep an eye on is that Whisper is strongly bound to processing a 30-second window at a time. So if you send it 30 seconds of audio, and it decodes it, then you send it another one second of audio, the only sensible way it can work is to have it reprocess seconds 2s-30s in addition to the new data at 31s. If there was a way to have it just process the update, then there's every possibility it could avoid…
[dead]
Re: Transcribro: On-device Accurate Speech-to-text
#53Earlier quoted context omitted.
It is not streaming in the way people normally use this term. It's a fuzzy notion but typically streaming means something encompassing: - Processing and emitting results on something closer to word by word level - Allowing partial results while the user is still speaking and mid-segment - Not relying on an external segmenter to determine the chunking (and therefore also latency) of the output.
This is fascinating because if your hint in another comment indicates you worked on this at Google, it's entirely possible I have this all wrong because I'm missing the actual ML part - I wrote the client encoder & server decoder for Opus and the client-side UI for the SODA launch, and I'm honestly really surprised to hear Google has different stuff. The client-side code loop AGSA used is 100% replicated, in my exper…
You are missing the speech decoding part. I can't speak to why the clients you were working on were doing what they were doing. For a different reference point see the cloud streaming api.
This is a good public reference: https://research.google/blog/an-all-neural-on-device-speech-...
Possibly confusions from that doc: "RNN-T" is entirely orthogonal to RNNs (and not the only streamable model). Attention is also orthogonal to streaming. A chunked or sliding window attention can stream, a bi-directional RNN cannot. How you think of an encoder and a decoder streaming is also different.
At a practical level, if a model is fast enough, and VAD is doing an adequate job, you can get something that looks like "streaming" which a non-streaming model. If a streaming model has tons of look-ahead or a very large input chunk size, its latency may not feel a lot better.
Where the difference is sharp is where VAD is not adequate: Users speak in continuous streams of audio, they leave in unusual gaps within sentences and run sentences together. A non-streaming system either hurts quality because sentences (or even words) get broken up that shouldn't, or has to wait forever and doesn't get a chance to run, when a streaming system would have already been producing output.
And to your points about echo cancellation and interference: There's many text only operations that benefit from being able to start early in the audio stream, not late.
I just went through process of helping someone stand up an interactive system with whisper etc and the lack of an open sourced whisper-quality streaming system is such a bummer because it really is so much laggier than it has to be.
Re: Transcribro: On-device Accurate Speech-to-text
#54Earlier quoted context omitted.
iOS already has on-device dictation built into the standard keyboard. Years ago it got sent to the cloud, but as long as you have an iPhone from the past few years it's on-device.
You're right that it exists, but it's complete crap outside a quiet environment. Try to use it while walking around outside or in any semi-noisy area and it fails horribly (iPhone 13, so YMMV if you have a newer one). You cannot use an iPhone as a dictation device without reviewing the transcribed text, which IMO defeats the purpose of dictation. Meanwhile, i've gotten excellent results on the iPhone from a Whipser->…
I've never found real-time dictation software that doesn't need to be reviewed.
I'm definitely waiting for Apple to upgrade their dictation software to the next generation -- I have my own annoyances with it -- but I haven't found anything else that works way better, in real time, on a phone, that runs in the background (like as part of the keyboard).
You talk about Whisper but that doesn't even work in real time, much less when you have to run it through an LLM.
Re: Transcribro: On-device Accurate Speech-to-text
#55[dead]
Re: Transcribro: On-device Accurate Speech-to-text
#56Earlier quoted context omitted.
FUTO and Transcribro are open source.
No, FUTO made a new "Source First License"[1] that is not Open Source by the OSI definition. [1] https://github.com/futo-org/android-keyboard/blob/master/LIC...
I can get behind people doing their own custom "licenses" that amount to throwing their work into the public domain, but if someone builds their own limited licenses around a thing, I won't touch their product. This FUTO license is garbage. Use a real license and either be open source or not; inventing new personal licenses doesn't do anyone any good.
Re: Transcribro: On-device Accurate Speech-to-text
#57Not sure what I'm doing wrong, but I tried installing it on a GrapheneOS device with Play Services installed and nothing happened. When I pushed the mic button, it changed to look pressed for a second, and went back to normal. Nothing happened when I spoke. Tried holding it down while speaking. Still nothing.
I'm very interested in using this, but I can't even find a way to try to troubleshoot it. I'm not finding usage instructions, never mind any kind of error messages. It just doesn't do anything.
This is especially interesting to me because the screenshot on the repo is from Vanadium, which strongly suggests to me that it's from a GrapheneOS device itself.
Re: Transcribro: On-device Accurate Speech-to-text
#58Not sure what I'm doing wrong, but I tried installing it on a GrapheneOS device with Play Services installed and nothing happened. When I pushed the mic button, it changed to look pressed for a second, and went back to normal. Nothing happened when I spoke. Tried holding it down while speaking. Still nothing. I'm very interested in using this, but I can't even find a way to try to troubleshoot it. I'm not finding usa…
You're correct I do use GrapheneOS. Hm do you have the global microphone toggle off? There's an upstream issue that causes SpeechRecognizer implementations to silently fail when the microphone toggle is off. You may have to force-stop Transcribro after turning it on.
Re: Transcribro: On-device Accurate Speech-to-text
#59Not sure what I'm doing wrong, but I tried installing it on a GrapheneOS device with Play Services installed and nothing happened. When I pushed the mic button, it changed to look pressed for a second, and went back to normal. Nothing happened when I spoke. Tried holding it down while speaking. Still nothing. I'm very interested in using this, but I can't even find a way to try to troubleshoot it. I'm not finding usa…
You're correct I do use GrapheneOS. Hm do you have the global microphone toggle off? There's an upstream issue that causes SpeechRecognizer implementations to silently fail when the microphone toggle is off. You may have to force-stop Transcribro after turning it on. https://github.com/soupslurpr/Transcribro/issues/3
I didn't think I did, but cycling it a couple times and restarting did fix! Great guess!
The thing I'm tripping over now is just that I keep pressing the button more than once when I'm done speaking because it's not clear that it registered the first time. If it could even just stay "pressed" or something while it processes the text, I think that would make it clearer. Any third state for the button would do I think.
Looking forward to using this! Thanks!
Re: Transcribro: On-device Accurate Speech-to-text
#60Earlier quoted context omitted.
You're correct I do use GrapheneOS. Hm do you have the global microphone toggle off? There's an upstream issue that causes SpeechRecognizer implementations to silently fail when the microphone toggle is off. You may have to force-stop Transcribro after turning it on. https://github.com/soupslurpr/Transcribro/issues/3
I didn't think I did, but cycling it a couple times and restarting did fix! Great guess! The thing I'm tripping over now is just that I keep pressing the button more than once when I'm done speaking because it's not clear that it registered the first time. If it could even just stay "pressed" or something while it processes the text, I think that would make it clearer. Any third state for the button would do I think.…
Good to hear its working.
Ah, it currently uses the Jetpack Compose toggle button but I do suppose it does actually have three states instead of two. I initially wanted to add a loading circle inside the button but wasn't able to without messing up the padding and such.
Hope you enjoy using Transcribro!