Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

github.com

Post reply on HN
Print view View original

71–80 of 354 posts

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#71

Post by wongarsu »

baobabKoodaa wrote: post #52
wongarsu wrote: post #48

Earlier quoted context omitted.

And the German is “subtitles of [public broadcaster] for [content network], 2017 I'm not sure this is really overfitting, the network does exactly what the training data demands. According to the training data silence art the end transcribes to a copyright notice or subtitle credits

> I'm not sure this is really overfitting, the network does exactly what the training data demands. What do you think overfitting is, if not that?

Overfitting would be replicating overly specific details. Like if a specific pattern of silence (or quiet noise) matched to specific copyright notices.

But in this case the behavior seems to generalize over multiple languages, with the model choosing representative "outro silence" captions depending on the language. Which is consistent with the training data showing that outro silence is captioned.

If the model was generalizing perfectly it would show something like "[subtitle credits here]" but that'd be demanding a bit much.

Transcribing outro silence as silence despite the training data consistently transcribing outro silence differently from regular silence would be underfitting

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#72

Post by wongarsu »

samrus wrote: post #62
wongarsu wrote: post #48

Earlier quoted context omitted.

And the German is “subtitles of [public broadcaster] for [content network], 2017 I'm not sure this is really overfitting, the network does exactly what the training data demands. According to the training data silence art the end transcribes to a copyright notice or subtitle credits

fitting on noise in the training data is exactly what overfitting is. underfitting is smoothing out signal

Exactly. Underfitting would be if the model doesn't pick up on the fact that outro silence is labeled differently from regular silence and transcribes them the same

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#73

Post by wahnfrieden »

isoprophlex wrote: post #43
cyp0633 wrote: post #24

The same happens with whisper-large-v3 on Chinese transcription: silence is transcribed to something like "please upvote, share and favourite this video". I suspect they trained the model on some random YouTube video without carefully picking really useful data.

Indeed, with another model I would get persistent transcriptions of silent parts into 'Thanks for watching!' or '[MUSIC]'. Pretty dumb that this failure mode wasn't caught in some QA process, and there are now multiple transcription models suffering from the same issue. Having silent parts in your input audio seems like it should be a very common occurrence...

whisper MUST be combined with silence detection / VAD

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#74

Post by [deleted] »

bGl2YW5j wrote: post #60
stingraycharles wrote: post #30

Earlier quoted context omitted.

How is this overfitting, rather than a data quality / classification issue?

If the model was able to generalise, you’d expect it to output something like “[silence]” or “…”, in response to silence. Instead, it reverted to what it has seen before (in the training data), hence the overfit.

[deleted]

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#75

Post by maxbond »

wongarsu wrote: post #71
baobabKoodaa wrote: post #52

Earlier quoted context omitted.

> I'm not sure this is really overfitting, the network does exactly what the training data demands. What do you think overfitting is, if not that?

Overfitting would be replicating overly specific details. Like if a specific pattern of silence (or quiet noise) matched to specific copyright notices. But in this case the behavior seems to generalize over multiple languages, with the model choosing representative "outro silence" captions depending on the language. Which is consistent with the training data showing that outro silence is captioned. If the model was g…

The optimizer is functioning correctly, and the pattern really exists in the training data. But consider:

- This behavior damages the model's performance on out of sample data; every word you predict during silence increases the transcript's Word Error Rate.

- These translation credits are an artifact of our training data, and not a reflection of the process we are modeling (spoken language).

So, while you are correct about the mechanism at work here, it is still correct to call learning a spurious pattern which damages our performance "overfitting".

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#76

Post by xyproto »

isoprophlex wrote: post #70
inglor_cz wrote: post #67

Earlier quoted context omitted.

Yeah, I studied mathematics (algebra and number theory) and zero is the point, often sporting discontinuities, or weird asymptotic behavior. Quite a lot of algorithms use some form of division and zero is the only number in our typical structures (Z, Q, R, C), that cannot be used to divide with.

Well, now in this brave new age of AI we can enjoy computer programs crashing with an Error: division by please upvote, share and like!

This also works; I upvoted your comment.

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#77

Post by boredhedgehog »

Incipient wrote: post #31
actionfromafar wrote: post #28

Earlier quoted context omitted.

Haha, trained on torrented movies! :-D The MPA must be so proud.

It's absolutely insane that these companies can't be held liable for what is obvious piracy.

It's an indication how few people consider license infringements as a matter of actual moral import. Those tend to evoke strong feelings.

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#78

Post by wongarsu »

h784gljf wrote: post #22
MrGilbert wrote: post #14

Earlier quoted context omitted.

> I wonder if the ZDF gave its approval for it being used for LLM training though? I am pretty sure they didn't get asked.

Just like the people forced to pay for ZDF under threat of imprisonment.

I'm being made to pay for Autobahnen I barely use, finance kindergartens despite not having a child, and made to pay into public pensions with little hope of getting close to the same value out. All under threat of imprisonment, many without a way to even refuse (not that I'd want to) The only thing that sets the pubic broadcasting fee apart is that it's collected separately from taxes in an attempt to reduce the influence politicians have on broadcasters

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#79

Post by verzali »

tarikozket wrote: post #39

this happens in Turkish too. I believe the reason is that the movie subtitles were used for training without cleaning up the comments / intros subtitle authors leave in them. leaving personal comments, jokes, reactions, intros in subtitles is very common in eastern cultures. Turkish readers will probably remember “esekadam iyi seyirler diler” :)

That name translates as "Donkey Man" btw :D
Post reply on HN