Whisper V3 is released! https://github.com/openai/whisper/commit/c5d42560760a05584c1... Looks like it's just a new checkpoint for the large model. It would be nice to have updates for the smaller models too. But it'll be easy to integrate with anything using Whisper V2. I'm excited to add it to my local voice AI ( https://www.microsoft.com/store/apps/9NC624PBFGB7 ) I assume ChatGPT voice has been using Whisper V3 and…
Do you also get those hallucinations just on silence? I kind of wonder if they had a bunch of training data of video with transcripts, but some of the video/audio was truncated and the transcript still said the last speech, and so now it thinks silence is just another way of signing off from a TV program. IMHO the bottleneck on voice now is all the infrastructure around it. How do you detect speech starting and stopp…
Silence is when you get the most hallucinations. But there is a trick supported by some implementations that helps a lot. Whisper does have a special token that it predicts for silence. You can look at the probability of that token even when it's not picked during sampling. Hallucinations often have a relatively high probability for the nospeech token compared to actual speech, so that can help filter them out.
As for all the surrounding stuff like detecting speech starting and stopping and listening for interruptions while talking, give my voice AI a try. It has a rough first pass at all that stuff, and it needs a lot of work but it's a start and it's fun to play with. Ultimately the answer is end-to-end speech-to-speech models, but you can get pretty far with what we have now in open source!