Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lingua-Go, the most accurate language detection for Go

github.com

Post reply on HN
Print view View original

21–26 of 26 posts

Re: Lingua-Go, the most accurate language detection for Go

#21

Post by Matumio »

blurker wrote: post #4
koeng wrote: post #2

I appreciate how direct and clear what this library does and who it is for. I have no need for it now, but after 1 paragraph of reading I’ll be remembering its name for later.

Yes that puts it really well and I agree. Too often I'll end up at a project where I'll still have little understanding of what it does or why it exists even after reading the readme. This project is super clear! But maybe that's what we should expect from a natural language library

No, it's not something you can take for granted. Ever heard of Vowpal Wabbit? It's one of the big old libraries in this space. Now read the first sentence of its landing page and compare. Good luck remembering what it does, or even just its name.

Re: Lingua-Go, the most accurate language detection for Go

#22

Post by akreal »

Xeoncross wrote: post #19

Great work, thanks for sharing! I see https://github.com/google/cld3 , but how does this compare with https://github.com/CLD2Owners/cld2 which is used by the large https://commoncrawl.org project to classify billions of samples from the whole internet?

There is also a comparison with CLD 2 in the repo of sister Python library:

https://github.com/pemistahl/lingua-py#4-how-good-is-it

CDL 2 seems to be slightly less accurate than CLD 3 on average.

Re: Lingua-Go, the most accurate language detection for Go

#23

Post by pemistahl »

doubtfuluser wrote: post #7

How does it actually compare to fasttext [1] in performance. Building an interface to that in GO shouldn’t be too complicated. The claim that all language identification (lid) relies on ngrams is bold and there has been a switch to pure neural network based approaches. [1] https://fasttext.cc/docs/en/language-identification.html

I've compared the Python implementation of Lingua with fasttext. Lingua performs clearly better. Look here: https://github.com/pemistahl/lingua-py#4-how-good-is-it

Re: Lingua-Go, the most accurate language detection for Go

#24

Post by pemistahl »

Hello everyone,

I'm the author of Lingua. Thank you for sharing my work and making it known in the NLP world.

Apart from the Go implementation, I've implemented the library in Kotlin, Python and Rust. Just take a look at my profile if you are interested: https://github.com/pemistahl

Re: Lingua-Go, the most accurate language detection for Go

#26

Post by blurker »

Matumio wrote: post #21
blurker wrote: post #4

Earlier quoted context omitted.

Yes that puts it really well and I agree. Too often I'll end up at a project where I'll still have little understanding of what it does or why it exists even after reading the readme. This project is super clear! But maybe that's what we should expect from a natural language library

No, it's not something you can take for granted. Ever heard of Vowpal Wabbit? It's one of the big old libraries in this space. Now read the first sentence of its landing page and compare. Good luck remembering what it does, or even just its name.

Haha yeah. Well, I did say *should*, but I actually meant it as a joke. Definitely not something I take for granted :)
Post reply on HN