Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

github.com

Post reply on HN
Print view View original

121–130 of 354 posts

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#121

Post by 0points »

IshKebab wrote: post #116
0points wrote: post #106

Interesting! I used whipser last year to attempt to build an audio transcription tool but gave up due to excessive amount of hallucinated output no matter what model I used. It would produce seemingly ok output until you started paying attention. One example, it insisted that Biggie Smalls sings "Puttin five carrots in my baby girl ear". (its "carats"). It's apparently not useful in transcription as it don't reason […

That example is not hallucination, it's just a homonym with insufficiently clear context for the model to disambiguate it.

I'm well aware mishearing "carots" as "carrots" is not a hallucination.

That's an example I gave after having used Whisper, the topic of discussion.

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#122

Post by pavon »

Incipient wrote: post #31
actionfromafar wrote: post #28

Earlier quoted context omitted.

Haha, trained on torrented movies! :-D The MPA must be so proud.

It's absolutely insane that these companies can't be held liable for what is obvious piracy.

Anthropic is going to trial over pirating books for training. The judge was pretty clear that even if training is fair use, the training material must be obtained legally.

These regurgitations combined with proof that a model is familiar with a work could be sufficient evidence to force discovery to determine if the work was pirated.

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#123

Post by zettabomb »

pferde wrote: post #81
wahnfrieden wrote: post #73

Earlier quoted context omitted.

whisper MUST be combined with silence detection / VAD

Ah, the good old "you're holding it wrong". What good is a speech recognition tool that literally hears imaginary voices?

Considering that if you DO use VAD (voice activity detection), it's the best open weights voice recognition model by a very wide margin, it's quite good. I'd be willing to be that commercial products that "don't have this problem" are using VAD as well, and that this is well known to them. But Whisper is just the weights, and I suppose a simple reference implementation, not a full product.

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#124

Post by arblor »

Just to add some trivia: ChatGpt interprets(/ed) silence as "Sottotitoli e Revisione a cura di QTSS". Now many videos (mainly dailymotion) with autogenerated subtitles have their Transcripts full of the same message

i.e. https://www.dailymotion.com/video/x9g9d6u

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#125

Post by sofixa »

berkes wrote: post #88
vanschelven wrote: post #20

I wonder if hallucinated copyright claims (esp. like the ZDF one at the bottom of the OP) will be introduced as evidence in one of the court cases against "big AI"

Evidence against what? "Big AI" is transparent and open about the fact they use all sorts of copyrighted material to train the data. How would "we see an exact chunk of text from our copyrighted material" add to that?

Their main defence is that it's fair use because it's transformative (like a human reading a book, getting inspired, and writing something of their own) and not a copypaste illegal distribution (like a human scanning that book and selling it themselves).

Having models hallucinate copyright notices shows that some content is being copypasted as is, which kind of goes against the transformative argument.

(Note: I think that trying to litigate AI with current copyright laws is weird. They were created before LLMs were even imagined, so of course they can't handle them clearly. New laws are needed around this, not trying to bend over backwards to think about what a lawmarker a century ago would have thought about how transformative a thing they couldn't have imagined is.)

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#126

Post by sofixa »

jdiff wrote: post #51
tarikozket wrote: post #39

this happens in Turkish too. I believe the reason is that the movie subtitles were used for training without cleaning up the comments / intros subtitle authors leave in them. leaving personal comments, jokes, reactions, intros in subtitles is very common in eastern cultures. Turkish readers will probably remember “esekadam iyi seyirler diler” :)

Kind of mindblowing considering who it is we're talking about. Of all companies, OpenAI couldn't be bothered to throw an LLM at this problem? Finding amorphously phrased but clearly recognizable needles in large numbers of haystacks seems like a patently perfect task for them.

Don't even need an LLM, a regex would have sufficed (I've used my fair share of community sourced subtitles, and comments are almost always in a different font, colour, between brackets, etc etc).

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#127

Post by KeplerBoy »

rollcat wrote: post #56
isoprophlex wrote: post #43

Earlier quoted context omitted.

Indeed, with another model I would get persistent transcriptions of silent parts into 'Thanks for watching!' or '[MUSIC]'. Pretty dumb that this failure mode wasn't caught in some QA process, and there are now multiple transcription models suffering from the same issue. Having silent parts in your input audio seems like it should be a very common occurrence...

When I was taught mathematics, the zero value was always considered the most important edge case. You prove something for N=0 (or N=1), then for N=M+1. It's even more important in audio DSP: processing near-zeroes can end up being extremely CPU intensive, look up denormal/subnormal floats.

Denormals are flushed to zero by default on most GPUs by the way.

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#128

Post by mmcwilliams »

cyp0633 wrote: post #24

The same happens with whisper-large-v3 on Chinese transcription: silence is transcribed to something like "please upvote, share and favourite this video". I suspect they trained the model on some random YouTube video without carefully picking really useful data.

Similar in the English model. Pretty clear they trained on YouTube videos where creators will put that in otherwise silent sections to ensure it shows up for people with CC on.

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#130

Post by indrora »

cyp0633 wrote: post #24

The same happens with whisper-large-v3 on Chinese transcription: silence is transcribed to something like "please upvote, share and favourite this video". I suspect they trained the model on some random YouTube video without carefully picking really useful data.

When YouTube began building automatic transcriptions for captions, it regularly flagged any noise or music -- typically industrial noise -- with "[foreign]"

If it couldn't understand it, it was "foreign" for the longest time.

Post reply on HN