11–16 of 16 posts
Earlier quoted context omitted.
wouldn't something like 2-gram bayes be more accurate & faster?
I don't know. I haven't really done NLP before. My guess is that it would be slower, but not by a significant amount. It might be more accurate, so I could give it a shot.
It uses a slightly weird technique, though. Dictionary based and using a bloom filter for memory efficiency. Going forward, though, I plan to rewrite it to use a combination of n-grams and language "fingerprints."
I was also at Leeds Hack Day. I didn't know you were working on this until the presentations but I wrote a language detection library for Ruby some time back: https://github.com/peterc/whatlanguage It uses a slightly weird technique, though. Dictionary based and using a bloom filter for memory efficiency. Going forward, though, I plan to rewrite it to use a combination of n-grams and language "fingerprints."
Do you have any accuracy stats? I'm guessing my approach might work better in some cases because the models include frequency information too. Did you experience significant accuracy loss when adding new languages? Anyway, I'll run it over my test data and compare.
I was also at Leeds Hack Day. I didn't know you were working on this until the presentations but I wrote a language detection library for Ruby some time back: https://github.com/peterc/whatlanguage It uses a slightly weird technique, though. Dictionary based and using a bloom filter for memory efficiency. Going forward, though, I plan to rewrite it to use a combination of n-grams and language "fingerprints."
That's pretty cool! One of my friends suggested to use dictionaries and bloom filters but I've wanted to build probabilistic language models. Do you have any accuracy stats? I'm guessing my approach might work better in some cases because the models include frequency information too. Did you experience significant accuracy loss when adding new languages? Anyway, I'll run it over my test data and compare.
I was also at Leeds Hack Day. I didn't know you were working on this until the presentations but I wrote a language detection library for Ruby some time back: https://github.com/peterc/whatlanguage It uses a slightly weird technique, though. Dictionary based and using a bloom filter for memory efficiency. Going forward, though, I plan to rewrite it to use a combination of n-grams and language "fingerprints."
That's pretty cool! One of my friends suggested to use dictionaries and bloom filters but I've wanted to build probabilistic language models. Do you have any accuracy stats? I'm guessing my approach might work better in some cases because the models include frequency information too. Did you experience significant accuracy loss when adding new languages? Anyway, I'll run it over my test data and compare.
No, but as you have noted, the method has the intrinsic property of being less accurate with fewer words and more accurate the longer the text. As my anticipated use was for documents over 10-20 words, this was OK. I expect the other techniques I outlined that I'm switching to to yield more accurate results across the board.
Earlier quoted context omitted.
Any chance of posting the source?
Yeah, I'll probably push it to github later this week.
