Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic
1–10 of 354 posts
Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic
#2Whisper frequently generates random credits. I guess they didn't curate the dataset much at the time.
Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic
#3to save you a lookup:
The Arabic text "رجمة نانسي قنقر" translates to English as: "Nancy Qanqar's translation" or "Translation by Nancy Qanqar"
"رجمة" means "translation" and "نانسي قنقر" is the name "Nancy Qanqar"
Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic
#4Whisper is unusable IMO because of the hallucinations. Widely documented. Removing silence from audio clips helps, but even then it will auto correct grammar, translating bilingual speech, etc. Improved in the latest audio models but not solved [1]
Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic
#5to save you a lookup: The Arabic text "رجمة نانسي قنقر" translates to English as: "Nancy Qanqar's translation" or "Translation by Nancy Qanqar" "رجمة" means "translation" and "نانسي قنقر" is the name "Nancy Qanqar"
And it seems to be because the training data is largely unofficial subtitles from movies. Which often have a string like "Translated by X" at the end of the movie which is often silent while credits roll.
Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic
#6Interesting that this happens even on large v3. I had once done a deep dive into STT and Whisper Large was the only model that could correctly transcribe Yann LeCun
(it was a Lex Friedman podcast), ever since I held the belief that it was the best STT model, this was over 2 years ago
Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic
#7Whisper is unusable IMO because of the hallucinations. Widely documented. Removing silence from audio clips helps, but even then it will auto correct grammar, translating bilingual speech, etc. Improved in the latest audio models but not solved [1] 1. https://news.ycombinator.com/item?id=43427376
That's the problem with raws large models, it should always be coupled with satellite small models and logic. It's (probably) easier to detect hallucinations using a traditional ML/DL model that can catch mismatches (it's easy to build a synthetic dataset for this) than transcribing. And the simplest piece of code can detect a silence and that it should match no text.
Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic
#8to save you a lookup: The Arabic text "رجمة نانسي قنقر" translates to English as: "Nancy Qanqar's translation" or "Translation by Nancy Qanqar" "رجمة" means "translation" and "نانسي قنقر" is the name "Nancy Qanqar"
In Czech, Whisper usually transcribes music as "Titulky vytvořil JohnyX" ("subtitles made by JohnyX") for the same reason.
Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic
#9Classic overfitting
It's the LLM equivalent of thinking that an out-of-office reply is the translation: https://www.theguardian.com/theguardian/2008/nov/01/5
Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic
#10to save you a lookup: The Arabic text "رجمة نانسي قنقر" translates to English as: "Nancy Qanqar's translation" or "Translation by Nancy Qanqar" "رجمة" means "translation" and "نانسي قنقر" is the name "Nancy Qanqar"
And it seems to be because the training data is largely unofficial subtitles from movies. Which often have a string like "Translated by X" at the end of the movie which is often silent while credits roll.
make sense..