Curious if this suffers from the same hallucination problems as Chirp or not. For example, if you pass chirp some audio with noise or even no audio, it will barf text at you like "I don't know. I don't know. I don't know." until a request timeout fires after like 10 minutes. It's... really bad. For this reason, we've moved back to Whisper for timestamp accuracy and then Gemini Flash for transcription correction.
Gemini-3.5-Transcribe
11–20 of 140 posts
Re: Gemini-3.5-Transcribe
#12I am not sure if "Word Error Rate" captures what has always been wrong with transcription. My biggest complaint is that it inserts sentence breaks in random places, then fails to evaluate the result, even though it is obviously wrong. Then I have to go fix it which can be harder than having just typed it myself, due to the difficulty of positioning the Android cursor, the fact that it automatically capitalizes if you…
have another model do a pass to clean it up, saw a demo of local STT where someone did this, can fix a lot of things, especially with gotchas for the STT model in a clean-transcript.md
Re: Gemini-3.5-Transcribe
#13Curious if this suffers from the same hallucination problems as Chirp or not. For example, if you pass chirp some audio with noise or even no audio, it will barf text at you like "I don't know. I don't know. I don't know." until a request timeout fires after like 10 minutes. It's... really bad. For this reason, we've moved back to Whisper for timestamp accuracy and then Gemini Flash for transcription correction.
I am using it all day as a main dictation model, and I can say it's the best model in terms of accuracy, latency, and formatting I have ever used.
Re: Gemini-3.5-Transcribe
#14Hopefully YouTube automatic captions improve with this
Re: Gemini-3.5-Transcribe
#15I am not sure if "Word Error Rate" captures what has always been wrong with transcription. My biggest complaint is that it inserts sentence breaks in random places, then fails to evaluate the result, even though it is obviously wrong. Then I have to go fix it which can be harder than having just typed it myself, due to the difficulty of positioning the Android cursor, the fact that it automatically capitalizes if you…
have another model do a pass to clean it up, saw a demo of local STT where someone did this, can fix a lot of things, especially with gotchas for the STT model in a clean-transcript.md
I think the model can even evaluate itself. If it looks afterward at an output like "do you. Want to get lunch?" in the absence of affirmative evidence that the user wanted it that way, it should be able to see that it goofed.
Re: Gemini-3.5-Transcribe
#16Where does the compute happen?
I suppose it's a cloud thing?
Re: Gemini-3.5-Transcribe
#17I am not sure if "Word Error Rate" captures what has always been wrong with transcription. My biggest complaint is that it inserts sentence breaks in random places, then fails to evaluate the result, even though it is obviously wrong. Then I have to go fix it which can be harder than having just typed it myself, due to the difficulty of positioning the Android cursor, the fact that it automatically capitalizes if you…
have another model do a pass to clean it up, saw a demo of local STT where someone did this, can fix a lot of things, especially with gotchas for the STT model in a clean-transcript.md
I haven't tried it, but this looked promising for that exact task: https://huggingface.co/superwhisper/s1-mini
Re: Gemini-3.5-Transcribe
#18Where does the compute happen? I suppose it's a cloud thing?
Yep.
Re: Gemini-3.5-Transcribe
#19Re: Gemini-3.5-Transcribe
#20I've been testing it on Pixel 11 Pro and I mostly dislike it. It is convenient when you have something long to say without thinking about it first.
But the main issue is when you want to say something precise with specific wording it might "simplify" it and break the meaning.
Something like "I hesitated to check it, I should have verified" => "I should have verified" (The "I hesitated..." is removed but I said it because I wanted to let the person know that I thought about it earlier)
I tried the sentence few times and it always removed the first part.