Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No Language Left Behind

ai.facebook.com

Post reply on HN
Print view View original

41–50 of 166 posts

Re: No Language Left Behind

#41

Post by jw4ng »

btheshoe wrote: post #2

I'm not entirely sure why low resource languages are seen as such a high priority for AI research. It seems that by definition there's little payoff to solving translation for them.

We think it's important for AI to truly support everyone in the world. A world where AI only serves a subset of the population is not ideal. In machine translation, this means supporting as many language as possible at high quality. We also imagine a future where anyone will be able to communicate with anyone else seamlessly; this also means solving translations for all languages.

Re: No Language Left Behind

#42

Post by maestrae »

hello_im_angela wrote: post #37
maestrae wrote: post #32

Earlier quoted context omitted.

hey, this sounds silly but I can't seem to find a link of all the languages covered in the 200 hundred languages. I've looked at the website and the blogpost and neither have a readily available link. Seems like a major oversight. There is of course a drop down in both but the languages there are a lot less than 200. I'm particularly interested in a list of the 55 African languages for example.

We have a full list here (copy pastable): https://github.com/facebookresearch/flores/tree/main/flores2... and Table 1 of our paper ( https://research.facebook.com/publications/no-language-left-... ) has a complete list as well.

thank you!

Re: No Language Left Behind

#43

Post by bvanderveen »

Great! Facebook no longer have to provide content moderation in all the various corners of the world where they could accidentally enable the dissemination of misinformation and hate speech in minority languages. They can simply transform it into English and run it back through the existing moderation tooling!

Understanding foreign culture is about reading automated translations of online comments into your native language. It has nothing to do with putting the effort into learning a language and understanding the nuances and current events and issues of the culture it embeds.

The ESL (English as a single language) speakers over at Facebook don't even need to understand foreign cultures, because they already know everyone in the world needs to spend their lives staring into the Metaverse. So grateful that they are working on the world's fattest pipeline for exporting Anglophone culture to every corner of the planet!

Re: No Language Left Behind

#44

Post by protomyth »

jw4ng wrote: post #21
kwhitefoot wrote: post #8

What is a "low resource language"?

hey there, I work on this project. We categorize a language as low-resource if there are fewer than 1M publicly available, de-duplicated bitext samples. also see section 3, table 1 in the paper: https://research.facebook.com/publications/no-language-left-...

Looking at the list, I see a lack of Native American languages. Did anyone try to contact the tribes during this?

Re: No Language Left Behind

#45

Post by goodside »

hello_im_angela wrote: post #37
maestrae wrote: post #32

Earlier quoted context omitted.

hey, this sounds silly but I can't seem to find a link of all the languages covered in the 200 hundred languages. I've looked at the website and the blogpost and neither have a readily available link. Seems like a major oversight. There is of course a drop down in both but the languages there are a lot less than 200. I'm particularly interested in a list of the 55 African languages for example.

We have a full list here (copy pastable): https://github.com/facebookresearch/flores/tree/main/flores2... and Table 1 of our paper ( https://research.facebook.com/publications/no-language-left-... ) has a complete list as well.

Nice to see Esperanto made the cut — the only artificial language to do so, AFAICT.

Re: No Language Left Behind

#46

Post by protomyth »

Tabular-Iceberg wrote: post #27

My concern with this is that in low resource languages the unavoidable biases of the ML models might overpower their own organic development. We shrug off all the little quirks of machine translated text because it usually gets the point across, and we recognize them as quirks because most of what we read was written by real people with no such quirks. But when most of what you read contain those quirks, I fear those…

I think it will interesting when it runs into a language (e.g. Dakota) where the women and men speak differently. Should be an interesting test.

Re: No Language Left Behind

#47

Post by macintux »

froskur wrote: post #17
btheshoe wrote: post #6

Earlier quoted context omitted.

yes, but what principles justify the importance placed on low resource languages?

Low resource in this context means that there are few resources available to train a neural network with, not that there are few speakers. Although many low resource languages have relatively few speakers, there are also ones with tens of millions of speakers. The reason for emphasis is in my opinion twofold: 1) Allowing these people to use the fancy language technology in their own language is good in and of itself.…

Plus presumably we learn more from solving harder problems, and we prepare for one day needing to translate some alien language in a hurry.

Re: No Language Left Behind

#48

Post by robocat »

pesenti wrote: post #7

Blog post: https://ai.facebook.com/blog/nllb-200-high-quality-machine-t... Paper: https://research.facebook.com/publications/no-language-left-... Github: https://github.com/facebookresearch/fairseq/tree/nllb/

Also note comments from hello_im_angela (= Angela Fan) and jw4ng (= Jeff Wang). Those are the HN accounts for Angela and Jeff from No Languages left Behind.

Re: No Language Left Behind

#50

Post by microtherion »

As a native Swiss German speaker, my native language is not only low resource in general, but has the additional difficulty of not having a standardized orthography (many native speakers will exclusively write in Standard German, and use Swiss German only for spoken communication).

So you have a language with some economic opportunity (a few million speakers in a fairly wealthy country) but no clearly defined written interface, and an ambivalent attitude of many speakers towards the very idea of writing the language.

Post reply on HN