Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta's Omnilingual MT for 1,600 Languages

ai.meta.com

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Meta's Omnilingual MT for 1,600 Languages

#31

Post by dhosek »

ks2048 wrote: post #29
intended wrote: post #26

Earlier quoted context omitted.

There’s many nation states working on this, have you looked into availability of those data sets? What languages are you prioritizing?

Yes, there are government datasets, languge "acadamies" (or "regulators") - organizations focused on preserving / teaching the language, and often smaller, local publishers that publish material in their local language. I'm living in Guatemala, so have been focusing on the Mayan languages here (22 languages, millions of speakers).

As an aside, I remember visiting Guatemala (in the border area near Chiapas) in the early 90s and discovering that “Mayan” was not the monolith that I had been led to believe by my culturally narrow American education, but was a diverse collection of related cultures with multiple languages.

In one of the villages we visited, there was a language school where foreigners were learning Jacalteco. One student was from Israel and where most of the students had vocabulary lists in three columns (Jacalteco - Spanish - English), his had four columns where he did one more step of translation to Hebrew.

Re: Meta's Omnilingual MT for 1,600 Languages

#32

Post by BalinKing »

canjobear wrote: post #28
garyclarke27 wrote: post #15

They can translate 1600 languages, but they cannot do basic text formatting, where are the paragraphs?

It's an abstract for a paper, so it's officially supposed to be one paragraph.

In the paper itself, the abstract actually does have a paragraph break, so it's probably just an autoformatting issue or something.

Re: Meta's Omnilingual MT for 1,600 Languages

#33

Post by ccgreg »

ks2048 wrote: post #23

I'll be looking at this in detail. I've started a company to do similar things, https://6k.ai I'm currently concentrating on better data gathering for low-resource languages. When you look in detail at data like Common Crawl, finepdfs, and fineweb, (1) they are really lacking quality data sources if you know where to look, and (2) the sources they have are not processed "finely" enough (e.g. finepdfs classify each pa…

Common Crawl has been running a low-resource language project for 1.5 years now -- it's a hard problem.

Re: Meta's Omnilingual MT for 1,600 Languages

#34

Post by gojomo »

Can translate between 1600 languages.

Can't achieve subject-verb agreement in 1st sentence of their English abstract.

Advances made through No Language Left Behind (NLLB) have demonstrated that high-quality machine translation (MT) scale to 200 languages.

Re: Meta's Omnilingual MT for 1,600 Languages

#35

Post by omneity »

ks2048 wrote: post #23

I'll be looking at this in detail. I've started a company to do similar things, https://6k.ai I'm currently concentrating on better data gathering for low-resource languages. When you look in detail at data like Common Crawl, finepdfs, and fineweb, (1) they are really lacking quality data sources if you know where to look, and (2) the sources they have are not processed "finely" enough (e.g. finepdfs classify each pa…

Hey, this is super cool! I’ve been working on a similar problem, focusing on low-resource and underserved languages including the Mayan family, and have published some research and open resources around that [0, 1].

On the data side, I’ve found that the biggest bottleneck isn’t collecting text (it’s out there!) but reliable language identification. It’s often difficult or ambiguous to separate languages cleanly in datasets like Common Crawl, Fineweb, or others. I worked on improving this a bit for Fineweb 2 for my native language, that might inspire you [3].

Many of the challenges you mention seem to recur across regions and language families, so I’d love to connect and compare notes sometime. Feel free to reach me at omar [at] the labs site below.

0: https://wikilangs.org

1: https://omneitylabs.com

2: https://huggingface.co/blog/omarkamali/gherbal-multilingual-...

Re: Meta's Omnilingual MT for 1,600 Languages

#39

Post by mandeepj »

omneity wrote: post #35
ks2048 wrote: post #23

I'll be looking at this in detail. I've started a company to do similar things, https://6k.ai I'm currently concentrating on better data gathering for low-resource languages. When you look in detail at data like Common Crawl, finepdfs, and fineweb, (1) they are really lacking quality data sources if you know where to look, and (2) the sources they have are not processed "finely" enough (e.g. finepdfs classify each pa…

Hey, this is super cool! I’ve been working on a similar problem, focusing on low-resource and underserved languages including the Mayan family, and have published some research and open resources around that [0, 1]. On the data side, I’ve found that the biggest bottleneck isn’t collecting text (it’s out there!) but reliable language identification. It’s often difficult or ambiguous to separate languages cleanly in da…

You both might find it useful - https://news.ycombinator.com/item?id=44950661

I’ve also recently started in this space: building an agent, for a client, who can communicate in multiple languages.

Post reply on HN