Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No Language Left Behind

ai.facebook.com

Post reply on HN
Print view View original

51–60 of 166 posts

Re: No Language Left Behind

#51

Post by hello_im_angela »

protomyth wrote: post #44
jw4ng wrote: post #21

Earlier quoted context omitted.

hey there, I work on this project. We categorize a language as low-resource if there are fewer than 1M publicly available, de-duplicated bitext samples. also see section 3, table 1 in the paper: https://research.facebook.com/publications/no-language-left-...

Looking at the list, I see a lack of Native American languages. Did anyone try to contact the tribes during this?

We interviewed speakers of low-resource languages from all over the world to understand the human need for this kind of technology --- what do people actually want, how would they use it, and what's the quality they would find useful? Many low-resource languages lack data online, but are spoken by millions. However, many indigenous languages are spoken by smaller numbers of people, and we are definitely interested in partnering with local communities to co-develop technology and have been actively investigating these collaborations but don't have much to share yet.

Re: No Language Left Behind

#52

Post by hello_im_angela »

goodside wrote: post #45
hello_im_angela wrote: post #37

Earlier quoted context omitted.

We have a full list here (copy pastable): https://github.com/facebookresearch/flores/tree/main/flores2... and Table 1 of our paper ( https://research.facebook.com/publications/no-language-left-... ) has a complete list as well.

Nice to see Esperanto made the cut — the only artificial language to do so, AFAICT.

ha yes, that's correct. If you have thoughts on specific constructed languages where having translation would really help people, let us know!

Re: No Language Left Behind

#53

Post by Jabbles »

jw4ng wrote: post #38
dangom wrote: post #26

Earlier quoted context omitted.

What is the greatest insight you gained and could share with non-experts from working on this project?

I gained a deeper understanding of what it truly means to be inclusive. Every language is unique just like everybody and making sure content works for all and including as many people as possible is really really hard, but through this project i'm hopeful we are taking it one step further

> Every language is unique just like everybody

TBH it just sounds like you've redefined the word "unique".

Re: No Language Left Behind

#54

Post by albertzeyer »

Note that very recently Google has done something very similar: "Building Machine Translation Systems for the Next Thousand Languages": https://arxiv.org/abs/2205.03983 https://ai.googleblog.com/2022/05/24-new-languages-google-tr...

The Facebook paper has some direct comparison to that work.

Re: No Language Left Behind

#56

Post by rmbyrro »

microtherion wrote: post #50

As a native Swiss German speaker, my native language is not only low resource in general, but has the additional difficulty of not having a standardized orthography (many native speakers will exclusively write in Standard German, and use Swiss German only for spoken communication). So you have a language with some economic opportunity (a few million speakers in a fairly wealthy country) but no clearly defined written…

This only makes the problem behind the NLLB project even more interesting to solve

Re: No Language Left Behind

#57

Post by rmbyrro »

LtWorf wrote: post #49

Facebook translations are horrifying for the mainstream languages already. They go from completely wrong to kinda understandable but still wrong.

Looks like they're investing to get better. The model is also available and they called for contributions to improve it.

Re: No Language Left Behind

#58

Post by shuraih »

jw4ng wrote: post #19

Jeff Wang here with my fellow Meta AI colleague Angela Fan from No Languages left Behind, seeing the comments flowing through. If you want to ask us anything, go for it!

Hey Jeff, I’m a native speaker of Dhivehi — the language spoken by the people of Maldives. Since I couldn’t find a full list of supported languages I was wondering if Dhivehi is / would be integrated.

Re: No Language Left Behind

#59

Post by hello_im_angela »

microtherion wrote: post #50

As a native Swiss German speaker, my native language is not only low resource in general, but has the additional difficulty of not having a standardized orthography (many native speakers will exclusively write in Standard German, and use Swiss German only for spoken communication). So you have a language with some economic opportunity (a few million speakers in a fairly wealthy country) but no clearly defined written…

sooo real. Many low-resource languages have many different natural variants, can be written in multiple scripts, don't have as much written standardization, or are mainly oral. As part of the creation of our benchmark, FLORES-200, we tried to support languages in multiple scripts (if they are naturally written like that) and explored translating regional variants (such as Moroccan Arabic, not just Arabic).

As an aside, the question of how to think about language standardization is really complex. We wrote some thoughts in Appendix A of our paper: https://research.facebook.com/publications/no-language-left-...

Re: No Language Left Behind

#60

Post by zen_1 »

protomyth wrote: post #46
Tabular-Iceberg wrote: post #27

My concern with this is that in low resource languages the unavoidable biases of the ML models might overpower their own organic development. We shrug off all the little quirks of machine translated text because it usually gets the point across, and we recognize them as quirks because most of what we read was written by real people with no such quirks. But when most of what you read contain those quirks, I fear those…

I think it will interesting when it runs into a language (e.g. Dakota) where the women and men speak differently. Should be an interesting test.

Doesn't seem to be a big issue for Arabic, where verbs are gendered (so in the sentence "I am going to the store", the verb "to go" will be either masculine or feminine, reflecting the speaker's gender).
Post reply on HN