21–30 of 46 posts
The tricky parts are usually political - are users going to be angry if you confuse Indonesian with Malaysian, or so on?
Alternatively, people can just download langid.py[1] and do language detection locally. This is not a particularly hard problem - I think it's doable by undergrad ML or NLP classes. The tricky parts are usually political - are users going to be angry if you confuse Indonesian with Malaysian, or so on? [1] https://github.com/saffsd/langid.py
In fact, we had a course for high school students where they learnt how a language guesser works and where they had to change a language guesser. A simplistic method that already works very well is:
* Create an n-gram fingerprint for each language by making a list of character uni-, bi-, and trigrams ordered by their frequency in a text. Retain the (say) 300 most frequent n-grams.
* To categorize a text, create a fingerprint for that text. Then compute for each language the sum n-gram rank differences. If an n-gram does not occur, the difference is the fingerprint size. Finally, pick the language with the lowest sum.
Of course, you can do fancier things, such as training a SVM or logistic regression classifier with n-grams and words as features, etc.
An interesting variation is to be able to distinguish different languages in a text. E.g. a Dutch text with English quotes.
Alternatively, people can just download langid.py[1] and do language detection locally. This is not a particularly hard problem - I think it's doable by undergrad ML or NLP classes. The tricky parts are usually political - are users going to be angry if you confuse Indonesian with Malaysian, or so on? [1] https://github.com/saffsd/langid.py
I think it's doable by undergrad ML or NLP classes. In fact, we had a course for high school students where they learnt how a language guesser works and where they had to change a language guesser. A simplistic method that already works very well is: * Create an n-gram fingerprint for each language by making a list of character uni-, bi-, and trigrams ordered by their frequency in a text. Retain the (say) 300 most fr…
Hmm, it takes 5+ seconds to get a response, and it chokes on the same test phrase as Google, thinking "Ik hou van vette lettertypes." is Norwegian...
And it looks like that they are using the following library: http://code.google.com/p/language-detection/ - at least the number & list of languages is very similar :-)
As others already mentioned, it would be good to have users try examples before signup.
I see three spelling errors in your language list:
- Panjabi should be Punjabi;
- Teligu should be Telugu;
- Ukraininan should be Ukrainian.
There are also a few grammar problems earlier in the document, and style problems (e.g. English doesn't use a space before sentence-ending punctuation marks).
Earlier quoted context omitted.
I think it's doable by undergrad ML or NLP classes. In fact, we had a course for high school students where they learnt how a language guesser works and where they had to change a language guesser. A simplistic method that already works very well is: * Create an n-gram fingerprint for each language by making a list of character uni-, bi-, and trigrams ordered by their frequency in a text. Retain the (say) 300 most fr…
It's easy to write a language guesser, but's not easy to write a good one. Even Google Translate is not prefect (see below).
There's a multitude of problems with real-world texts that a robust guesser must deal with gracefully: short texts; texts in none of the languages the "guesser" was trained for (is it able to return "none of the above?" or does it return a random one then?); texts in multiple languages (incl. common noun phrases phrases inserted into text in another language); texts with parts repeated multiple times (web pages and blogs in particular are a bitch!), which skews char/word distributions and messes up statistical models etc.
It's the same thing as with spelling correction, really. "But Norvig did it in 1.5 lines of Python!" See "A Spellchecker Used To Be A Major Feat of Software Engineering" at https://news.ycombinator.com/item?id=3466927 Spoiler: it still is, except for "drive-by ML apps".