I wonder how this performs on short text posts like tweets. At my last gig where we did social media text analysis we used a few different packages (chromium, guess-language, and our own ngram classifier) and still had pretty low accuracy for tweets.
Show HN: Language detection as a service
31–40 of 46 posts
Re: Show HN: Language detection as a service
#32curl -XPOST -d 'hello' 'https://getlang.io/get?token=...'
{
"code": "fi",
"name": "suomi, suomen kieli",
"name_en": "Finnish"
}
O_O
Re: Show HN: Language detection as a service
#33"test it out" comes back as french...
Maybe you've fallen in the 1% error rate ?
Language guessing is rather hard when few letters are used especially if you use statistical methods. I think after 20 something letters you enter >%95 accuracy zone. In a simple library I wrote ( https://github.com/ahmetaa/zemberek-nlp/tree/master/lang-id Works for 60 languages but no docs yet) , for Turkish and English test results are:
For 20 letters
TR=95.90 EN=94.96
For 50 Letters
TR=99.44 EN=99.53
If 50 letters are used in a Doc, it identifies about 20000 docs per second in a decent desktop.
Re: Show HN: Language detection as a service
#34I wonder how this performs on short text posts like tweets. At my last gig where we did social media text analysis we used a few different packages (chromium, guess-language, and our own ngram classifier) and still had pretty low accuracy for tweets.
Have you look at the metadata returned by a tweet? They also returned language, as well as location of the tweeter, which gives you some clues.
Re: Show HN: Language detection as a service
#35Alternatively, people can just download langid.py[1] and do language detection locally. This is not a particularly hard problem - I think it's doable by undergrad ML or NLP classes. The tricky parts are usually political - are users going to be angry if you confuse Indonesian with Malaysian, or so on? [1] https://github.com/saffsd/langid.py
I think it's doable by undergrad ML or NLP classes. In fact, we had a course for high school students where they learnt how a language guesser works and where they had to change a language guesser. A simplistic method that already works very well is: * Create an n-gram fingerprint for each language by making a list of character uni-, bi-, and trigrams ordered by their frequency in a text. Retain the (say) 300 most fr…
"An interesting variation is to be able to distinguish different languages in a text. E.g. a Dutch text with English quotes."
Do you know any interesting work related to the language distinction idea on the same text?
Re: Show HN: Language detection as a service
#36Earlier quoted context omitted.
I think it's doable by undergrad ML or NLP classes. In fact, we had a course for high school students where they learnt how a language guesser works and where they had to change a language guesser. A simplistic method that already works very well is: * Create an n-gram fingerprint for each language by making a list of character uni-, bi-, and trigrams ordered by their frequency in a text. Retain the (say) 300 most fr…
It's easy to write a language guesser, but's not easy to write a good one. Even Google Translate is not prefect (see below).
It's easy to write a language guesser, but's not easy to write a good one.
Obviously, it is highly domain and text length dependent (as I also mentioned in another comment).
But, e.g. Cavnar and Trenkle obtained a 99.8% accuracy on newsgroup articles in 14 languages using the method outlined above.
There are very few NLP tasks where you can achieve such high accuracy with relatively simple and understandable methods. That's why it is a nice subject for an NLP introduction to e.g. high school students.
I have worked in parsing and generation, where it is difficult to obtain satisfying results with many man years of work on newspaper text, let alone tweets or Youtube comments ;).
Re: Show HN: Language detection as a service
#37Earlier quoted context omitted.
It's easy to write a language guesser, but's not easy to write a good one. Even Google Translate is not prefect (see below).
Great point. Often overlooked by people who only know what I call "drive-by machine learning" (finished an online ML course or something). There's a multitude of problems with real-world texts that a robust guesser must deal with gracefully: short texts; texts in none of the languages the "guesser" was trained for (is it able to return "none of the above?" or does it return a random one then?); texts in multiple lang…
Often overlooked by people who only know what I call "drive-by machine learning" (finished an online ML course or something).
A bit sour, are we? ;)
The point is that it is an NLP task where it is relatively easy to get good results on general text (see Cavnar and Trenkle). So, it is a fun and satisfying exercise.
Saying there is difficult noisy data is pointing out the obvious ;).
Re: Show HN: Language detection as a service
#38Earlier quoted context omitted.
Great point. Often overlooked by people who only know what I call "drive-by machine learning" (finished an online ML course or something). There's a multitude of problems with real-world texts that a robust guesser must deal with gracefully: short texts; texts in none of the languages the "guesser" was trained for (is it able to return "none of the above?" or does it return a random one then?); texts in multiple lang…
Often overlooked by people who only know what I call "drive-by machine learning" (finished an online ML course or something). A bit sour, are we? ;) The point is that it is an NLP task where it is relatively easy to get good results on general text (see Cavnar and Trenkle). So, it is a fun and satisfying exercise. Saying there is difficult noisy data is pointing out the obvious ;).
If it's obvious to you, then you're not the target audience of my disclaimer :)
But HN responses to posts like these overwhelmingly suggest it's far from obvious.
So, it is a fun and satisfying exercise.
Re: Show HN: Language detection as a service
#39Earlier quoted context omitted.
I think it's doable by undergrad ML or NLP classes. In fact, we had a course for high school students where they learnt how a language guesser works and where they had to change a language guesser. A simplistic method that already works very well is: * Create an n-gram fingerprint for each language by making a list of character uni-, bi-, and trigrams ordered by their frequency in a text. Retain the (say) 300 most fr…
"An interesting variation is to be able to distinguish different languages in a text. E.g. a Dutch text with English quotes." Do you know any interesting work related to the language distinction idea on the same text?
I have never looked into that in detail. These may be some interesting leads:
http://mt-archive.info/IJCNLP-2008-Ehara.pdf http://202.41.85.68/knm-publications/lang_id_jql.pdf
Re: Show HN: Language detection as a service
#40how does this compare in accuracy to chromium's Compact Language Detector? https://code.google.com/p/chromium-compact-language-detector... https://github.com/mzsanford/cld
From my experience, the CLD works pretty well in the most cases. But you need to take care for encoding detection...
Yes, but you presumably need to get that right in order to encode as UTF-8 and send off to a third-party API...