Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

github.com

Post reply on HN
Print view View original

51–60 of 354 posts

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#51

Post by jdiff »

tarikozket wrote: post #39

this happens in Turkish too. I believe the reason is that the movie subtitles were used for training without cleaning up the comments / intros subtitle authors leave in them. leaving personal comments, jokes, reactions, intros in subtitles is very common in eastern cultures. Turkish readers will probably remember “esekadam iyi seyirler diler” :)

Kind of mindblowing considering who it is we're talking about. Of all companies, OpenAI couldn't be bothered to throw an LLM at this problem? Finding amorphously phrased but clearly recognizable needles in large numbers of haystacks seems like a patently perfect task for them.

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#52

Post by baobabKoodaa »

wongarsu wrote: post #48
hsn915 wrote: post #33

Earlier quoted context omitted.

ُThe Arabic text is the translator's self credit "Translated by Nancy Qanfar"

And the German is “subtitles of [public broadcaster] for [content network], 2017 I'm not sure this is really overfitting, the network does exactly what the training data demands. According to the training data silence art the end transcribes to a copyright notice or subtitle credits

> I'm not sure this is really overfitting, the network does exactly what the training data demands.

What do you think overfitting is, if not that?

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#53

Post by xg15 »

Yeah, the subtitle "credits" occur very frequently. I found with whisper-2, they're also triggered by music.

I suppose the cause is the same, generally subtitle creators adding all kinds of stuff during the credits that is NOT a transcript.

Seems to me it could have been filtered out relatively easily during training, by clipping the first and last few minutes of all audios. But I guess that's just in hindsight.

Whisper also likes to transcribe cut off speech or unintelligible noise as "Thank you". I have no idea where that is coming from, but I guess it's a very polite model...

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#54

Post by jdiff »

1718627440 wrote: post #49
jdiff wrote: post #45

Earlier quoted context omitted.

That's the magic of money. Download your favorite artist's discography for personal use? If the MPAA had its way (and it occasionally has), torrenting that could bankrupt you. The AI industry - soaking up every bit of media available online for commercial purposes, often reproducing it nearly identically - has enough money and capital to influence things its way. And only its way, in case anyone was hoping this might…

The movie industry also has some money and lobbying power. Surely this is a way larger threat than any single torrenter could ever be?

The fact that this is propping up the entire AI industry adds additional weight. When legislating or deciding court cases, some won't be willing to pop the cash cow, some will be worried about falling behind countries that don't enforce copyright evenly. IP owners are trying to go after the AI industry, with only mixed to poor success.

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#55

Post by darkwater »

unusual-name wrote: post #25
rob74 wrote: post #11

Earlier quoted context omitted.

Looks like they used more official sources for German - there, silence is apparently hallucinated as "Untertitelung des ZDF für funk, 2017" according to one of the comments on the issue. Which makes sense, as the public broadcasters' "Mediathek" is probably the largest freely available resource of subtitled videos in Germany. I wonder if the ZDF gave its approval for it being used for LLM training though?

Most content from Funk (youtubers funded by public german broadcasters) is available on youtube without any geoblocking or other limitations.

from the link[1] another user posted:

> We have a public service mandate, which means that we have very clear responsibilities according to the state media treaty. For us, this means that our top priority is actually reaching our target audience, namely approximately 15 million people living in Germany between the age of 14 and 29 who have internet access

It's not a binding contract for sure but I don't think that OpenAI or other AI scraper is their target.

[1] https://presse.funk.net/das-ist-funk/

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#56

Post by rollcat »

isoprophlex wrote: post #43
cyp0633 wrote: post #24

The same happens with whisper-large-v3 on Chinese transcription: silence is transcribed to something like "please upvote, share and favourite this video". I suspect they trained the model on some random YouTube video without carefully picking really useful data.

Indeed, with another model I would get persistent transcriptions of silent parts into 'Thanks for watching!' or '[MUSIC]'. Pretty dumb that this failure mode wasn't caught in some QA process, and there are now multiple transcription models suffering from the same issue. Having silent parts in your input audio seems like it should be a very common occurrence...

When I was taught mathematics, the zero value was always considered the most important edge case. You prove something for N=0 (or N=1), then for N=M+1.

It's even more important in audio DSP: processing near-zeroes can end up being extremely CPU intensive, look up denormal/subnormal floats.

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#57

Post by mort96 »

stingraycharles wrote: post #30
dlcarrier wrote: post #9

Classic overfitting It's the LLM equivalent of thinking that an out-of-office reply is the translation: https://www.theguardian.com/theguardian/2008/nov/01/5

How is this overfitting, rather than a data quality / classification issue?

Isn't overfitting just when the model picks up on an unintended pattern in the training data? Isn't that precisely what this is?

Re: Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic

#60

Post by bGl2YW5j »

stingraycharles wrote: post #30
dlcarrier wrote: post #9

Classic overfitting It's the LLM equivalent of thinking that an out-of-office reply is the translation: https://www.theguardian.com/theguardian/2008/nov/01/5

How is this overfitting, rather than a data quality / classification issue?

If the model was able to generalise, you’d expect it to output something like “[silence]” or “…”, in response to silence.

Instead, it reverted to what it has seen before (in the training data), hence the overfit.

Post reply on HN