Lingua-Go, the most accurate language detection for Go
1–10 of 26 posts
Re: Lingua-Go, the most accurate language detection for Go
#2I appreciate how direct and clear what this library does and who it is for. I have no need for it now, but after 1 paragraph of reading I’ll be remembering its name for later.
Re: Lingua-Go, the most accurate language detection for Go
#3> Most libraries only use n-grams of size 3 (trigrams) which is satisfactory for detecting the language of longer text fragments consisting of multiple sentences. For short phrases or single words, however, trigrams are not enough.
I only dabbled in language detection at a workshop at a conference years ago, but I was very impressed how well such models work on short text with only bigrams.
Maybe once you expand to over 70 languages does bi- and tri-grams fall short, but I just wanted to say that this is a usecase here very simple models can get you really far.
If you see a blog post where a language detection problem is solved with deep learning chances are the author doesn’t know what they are doing (towards datascience, I’m looking at you!) or it’s a tutorial for working with an NN framework.
Re: Lingua-Go, the most accurate language detection for Go
#4I appreciate how direct and clear what this library does and who it is for. I have no need for it now, but after 1 paragraph of reading I’ll be remembering its name for later.
Yes that puts it really well and I agree. Too often I'll end up at a project where I'll still have little understanding of what it does or why it exists even after reading the readme. This project is super clear! But maybe that's what we should expect from a natural language library
Re: Lingua-Go, the most accurate language detection for Go
#5> This engine first determines the alphabet of the input text and searches for characters which are unique in one or more languages. If exactly one language can be reliably chosen this way, the statistical model is not necessary anymore.
Can this be a problem? If a text in Language_A includes names/words of Language_B, only relying on special characters would wrongly classify the entire text as Language_B.
Re: Lingua-Go, the most accurate language detection for Go
#6What's a good way to detect languages in mixed-language passages? What's the state of art here?
For example, given "'I think, therefore I am' is the first principle of René Descartes's philosophy that was originally published in French as je pense, donc je suis.", is there a library that would tell me the main passage is in English, but contains fragments in French?
Re: Lingua-Go, the most accurate language detection for Go
#7How does it actually compare to fasttext [1] in performance.
Building an interface to that in GO shouldn’t be too complicated.
The claim that all language identification (lid) relies on ngrams is bold and there has been a switch to pure neural network based approaches.
[1] https://fasttext.cc/docs/en/language-identification.html
Re: Lingua-Go, the most accurate language detection for Go
#8> Most libraries only use n-grams of size 3 (trigrams) which is satisfactory for detecting the language of longer text fragments consisting of multiple sentences. For short phrases or single words, however, trigrams are not enough. I only dabbled in language detection at a workshop at a conference years ago, but I was very impressed how well such models work on short text with only bigrams. Maybe once you expand to o…
We're using libraries like this to try to guess the language of a book based on title alone (in case no other information is readily available), and trigram-based algorithms get it wrong often enough for it to be noticeable. I will look into replacing our current library with this one, it seems better suited for the task at hand.
Re: Lingua-Go, the most accurate language detection for Go
#9I see Dutch performs badly. I wouldn't be surprised if that's because of bad/noisy training data. Dutch web content contains an awful amount of English, which pollutes recognition. Cross-check the Dutch tokens with an English dictionary to be sure (although there is quite some overlap for frequent words, e.g. "is", "we", "are", "have", "bent", "had", "brief", etc., and rare ones like "keeshond").
BTW, the test statistic for recognizing individual words isn't interesting, unless you sample/weigh by word frequency.
Re: Lingua-Go, the most accurate language detection for Go
#10How does it actually compare to fasttext [1] in performance. Building an interface to that in GO shouldn’t be too complicated. The claim that all language identification (lid) relies on ngrams is bold and there has been a switch to pure neural network based approaches. [1] https://fasttext.cc/docs/en/language-identification.html
You can try with dataset that I’ve used to evaluate fasttext against different language detectors - it’s linked to this blog post: https://alexott.blogspot.com/2017/10/evaluating-fasttexts-mo...
I’ll try to find time to do it myself, but most probably only tomorrow