Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lingua-Go, the most accurate language detection for Go

github.com

Post reply on HN
Print view View original

11–20 of 26 posts

Re: Lingua-Go, the most accurate language detection for Go

#11

Post by alexott »

akie wrote: post #8
wodenokoto wrote: post #3

> Most libraries only use n-grams of size 3 (trigrams) which is satisfactory for detecting the language of longer text fragments consisting of multiple sentences. For short phrases or single words, however, trigrams are not enough. I only dabbled in language detection at a workshop at a conference years ago, but I was very impressed how well such models work on short text with only bigrams. Maybe once you expand to o…

We're using libraries like this to try to guess the language of a book based on title alone (in case no other information is readily available), and trigram-based algorithms get it wrong often enough for it to be noticeable. I will look into replacing our current library with this one, it seems better suited for the task at hand.

Yeah, language detection on short texts is quite complex. In my practice, N-grams doesn’t work well for them.

Re: Lingua-Go, the most accurate language detection for Go

#12

Post by yorwba »

kccqzy wrote: post #6

What's a good way to detect languages in mixed-language passages? What's the state of art here? For example, given "'I think, therefore I am' is the first principle of René Descartes's philosophy that was originally published in French as je pense, donc je suis." , is there a library that would tell me the main passage is in English, but contains fragments in French?

With an ngram-based model like this one, you can just feed it short substrings, since it doesn't take the larger context into account anyway. There'll be some problems at the boundary, because e.g. "as" is a word in both languages.

Re: Lingua-Go, the most accurate language detection for Go

#13

Post by yorwba »

tgv wrote: post #9

I see Dutch performs badly. I wouldn't be surprised if that's because of bad/noisy training data. Dutch web content contains an awful amount of English, which pollutes recognition. Cross-check the Dutch tokens with an English dictionary to be sure (although there is quite some overlap for frequent words, e.g. "is", "we", "are", "have", "bent", "had", "brief", etc., and rare ones like "keeshond"). BTW, the test statis…

My guess is that it has trouble distinguishing between Afrikaans and Dutch, Indonesian and Malay, and other similar pairs.

Re: Lingua-Go, the most accurate language detection for Go

#14

Post by hvdijk »

kccqzy wrote: post #6

What's a good way to detect languages in mixed-language passages? What's the state of art here? For example, given "'I think, therefore I am' is the first principle of René Descartes's philosophy that was originally published in French as je pense, donc je suis." , is there a library that would tell me the main passage is in English, but contains fragments in French?

Worth noting that this is on Lingua-Go's issues list for the 1.1.0 version: https://github.com/pemistahl/lingua-go/issues/9

Re: Lingua-Go, the most accurate language detection for Go

#15

Post by [deleted] »

rippeltippel wrote: post #5

> This engine first determines the alphabet of the input text and searches for characters which are unique in one or more languages. If exactly one language can be reliably chosen this way, the statistical model is not necessary anymore. Can this be a problem? If a text in Language_A includes names/words of Language_B, only relying on special characters would wrongly classify the entire text as Language_B.

[deleted]

Re: Lingua-Go, the most accurate language detection for Go

#17

Post by tgv »

yorwba wrote: post #13
tgv wrote: post #9

I see Dutch performs badly. I wouldn't be surprised if that's because of bad/noisy training data. Dutch web content contains an awful amount of English, which pollutes recognition. Cross-check the Dutch tokens with an English dictionary to be sure (although there is quite some overlap for frequent words, e.g. "is", "we", "are", "have", "bent", "had", "brief", etc., and rare ones like "keeshond"). BTW, the test statis…

My guess is that it has trouble distinguishing between Afrikaans and Dutch, Indonesian and Malay, and other similar pairs.

Overlap with those is not particularly large. n-grams are particularly sensitive to spelling, and e.g. Afrikaans writes "Hy het skool toe gegaan", whereas it would be "Hij is naar school gegaan" in Dutch.

Indonesian and Malay insert vowels in consonant clusters and replace quite a few consonants, so they should be easily distinguishable from Dutch, even on Dutch loan words (which are not that frequent anyway).

Dutch has a much larger overlap with German (probably the largest), but even those can be distinguished (by a human) with just a few words of a meaningful sentence. I find it difficult to come up with three words that could be a grammatical fragment in both languages, but even then I expect the n-gram frequencies to be quite diverging.

Re: Lingua-Go, the most accurate language detection for Go

#18

Post by yorwba »

tgv wrote: post #17
yorwba wrote: post #13

Earlier quoted context omitted.

My guess is that it has trouble distinguishing between Afrikaans and Dutch, Indonesian and Malay, and other similar pairs.

Overlap with those is not particularly large. n-grams are particularly sensitive to spelling, and e.g. Afrikaans writes "Hy het skool toe gegaan", whereas it would be "Hij is naar school gegaan" in Dutch. Indonesian and Malay insert vowels in consonant clusters and replace quite a few consonants, so they should be easily distinguishable from Dutch, even on Dutch loan words (which are not that frequent anyway). Dutch…

It's on single-word detection where the accuracy for Afrikaans and Dutch is between 50% and 60%. Understandable, considering "het" and "toe" are also Dutch words and "is" is also Afrikaans.

I meant that Indonesian and Malay would be difficult to distinguishing from each other.

Re: Lingua-Go, the most accurate language detection for Go

#20

Post by Radim »

wodenokoto wrote: post #3

> Most libraries only use n-grams of size 3 (trigrams) which is satisfactory for detecting the language of longer text fragments consisting of multiple sentences. For short phrases or single words, however, trigrams are not enough. I only dabbled in language detection at a workshop at a conference years ago, but I was very impressed how well such models work on short text with only bigrams. Maybe once you expand to o…

My experience is the opposite: character ngram models work "OK" on academic tasks and clean corpora. Not so much when unleashed on real data.

By "real", I mean texts in a mix of multiple languages (super common on the web); short texts; texts in a different (unknown) language where ngrams don't know how to say "I don't know" and return rubbish instead; texts in close languages; etc.

Going "deep learning" is not the only alternative. Even simpler methods can work significantly better, while being fully interpretable:

https://link.springer.com/chapter/10.1007/978-3-642-00382-0_...

Post reply on HN