Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini-3.5-Transcribe

blog.google

Post reply on HN
Print view View original

21–30 of 140 posts

Re: Gemini-3.5-Transcribe

#21

Post by Lucasoato »

I’ve tested at least 20 STT models in a benchmark I’ve set up with German, Italian and English voices from meetings in my company.

The voices contained very industry specific words, the languages changed from one sentence to another, sometimes words in a language were mentioned while a discussion was in another.

The only local model that satisfies me is Voxtral Mini 3b, the only paid API that is slightly better is eleven labs. Yes, Voxtral might not reach the best score in the benchmarks, but to me, it just solves a problem. It might not be the best in terms of speed... but that’s not a problem for me.

Happy to test this new model from Google but I’m not sure I’d go with that instead of something that can run so easily in my machine.

Re: Gemini-3.5-Transcribe

#22

Post by wolvoleo »

film42 wrote: post #11

Curious if this suffers from the same hallucination problems as Chirp or not. For example, if you pass chirp some audio with noise or even no audio, it will barf text at you like "I don't know. I don't know. I don't know." until a request timeout fires after like 10 minutes. It's... really bad. For this reason, we've moved back to Whisper for timestamp accuracy and then Gemini Flash for transcription correction.

I've seen Whisper do this exact thing as well. And also repeating a few words over and over near the end, words that were said but not completely at the end. It seems to overwrite the last few sentences with that so actual content is lost. It's really annoying especially because Whisper is really quite good otherwise. Sometimes a friend sends me audio messages from a noisy car and I can't make it all out but whisper gets it mostly right.

Re: Gemini-3.5-Transcribe

#24

Post by erulabs »

k9294 wrote: post #13
film42 wrote: post #11

Curious if this suffers from the same hallucination problems as Chirp or not. For example, if you pass chirp some audio with noise or even no audio, it will barf text at you like "I don't know. I don't know. I don't know." until a request timeout fires after like 10 minutes. It's... really bad. For this reason, we've moved back to Whisper for timestamp accuracy and then Gemini Flash for transcription correction.

I am using it all day as a main dictation model, and I can say it's the best model in terms of accuracy, latency, and formatting I have ever used.

mind if I ask what dictation tool you are using?

Re: Gemini-3.5-Transcribe

#25

Post by cnxhk »

Lucasoato wrote: post #21

I’ve tested at least 20 STT models in a benchmark I’ve set up with German, Italian and English voices from meetings in my company. The voices contained very industry specific words, the languages changed from one sentence to another, sometimes words in a language were mentioned while a discussion was in another. The only local model that satisfies me is Voxtral Mini 3b, the only paid API that is slightly better is el…

Any possibility to share some of the eval audio?

Re: Gemini-3.5-Transcribe

#26

Post by sync »

Crystalin wrote: post #20

I've been testing it on Pixel 11 Pro and I mostly dislike it. It is convenient when you have something long to say without thinking about it first. But the main issue is when you want to say something precise with specific wording it might "simplify" it and break the meaning. Something like "I hesitated to check it, I should have verified" => "I should have verified" (The "I hesitated..." is removed but I said it bec…

Are you in Smart or Verbatim mode?

https://ai.google.dev/gemini-api/docs/transcribe#transcripti...

Re: Gemini-3.5-Transcribe

#27

Post by sync »

film42 wrote: post #11

Curious if this suffers from the same hallucination problems as Chirp or not. For example, if you pass chirp some audio with noise or even no audio, it will barf text at you like "I don't know. I don't know. I don't know." until a request timeout fires after like 10 minutes. It's... really bad. For this reason, we've moved back to Whisper for timestamp accuracy and then Gemini Flash for transcription correction.

I eval'd this pretty heavily with no audio or just noise, as for us hallucinating a response is very bad. It works great in this scenario. There are some bugs, e.g. randomly exactly 20 seconds of silence will lead to a 403, but I'm sure these aspects will be improved over time. I didn't get any hallucinations though.

Re: Gemini-3.5-Transcribe

#29

Post by hinnisdael »

Lucasoato wrote: post #21

I’ve tested at least 20 STT models in a benchmark I’ve set up with German, Italian and English voices from meetings in my company. The voices contained very industry specific words, the languages changed from one sentence to another, sometimes words in a language were mentioned while a discussion was in another. The only local model that satisfies me is Voxtral Mini 3b, the only paid API that is slightly better is el…

Agree about Voxtral being the only model — local or cloud — that handles multilingual conversation really well. I‘m not sure what they do differently, but mixed-language sentences and industry terms don‘t seem to faze it where other model begin to struggle.

Re: Gemini-3.5-Transcribe

#30

Post by kridsdale1 »

Lucasoato wrote: post #21

I’ve tested at least 20 STT models in a benchmark I’ve set up with German, Italian and English voices from meetings in my company. The voices contained very industry specific words, the languages changed from one sentence to another, sometimes words in a language were mentioned while a discussion was in another. The only local model that satisfies me is Voxtral Mini 3b, the only paid API that is slightly better is el…

I use Parakeet 3. How does that fare in your testing?
Post reply on HN