11–20 of 39 posts
Naive Bayes is the former, me thinks.
Naive Bayes is also surprisingly powerful for deduplication of large datasets of messy data (and the related problem of record linkage). It's still competitive with cutting edge approaches if carefully applied, and usually much faster so it can be applied to big data. It's the engine behind Splink, the free Python library that I develop: https://github.com/moj-analytical-services/splink
Congrats on winning all those awards.
But in the world of the future which is likely look at the "LLM of everything spoken/written" and given you can solve P(Y|X) P(X|Y) from a model that can do P(Y, X); LLMs could just win because of integration with everything.
Naive Bayes is also surprisingly powerful for deduplication of large datasets of messy data (and the related problem of record linkage). It's still competitive with cutting edge approaches if carefully applied, and usually much faster so it can be applied to big data. It's the engine behind Splink, the free Python library that I develop: https://github.com/moj-analytical-services/splink
This is a very cool tool. I was testing fastlink recently but found some issues with the docs and implementation. I will check out your tool. Congrats on winning all those awards.
Fastlink was the inspiration for Splink - the fundamental statistical model is very similar. The first version of Splink was essentially a port to make it work faster and at greater scale, but we've subsequently added quite a bit of additional functionality
Feel free to ask questions if you run into any issues - we're usually fairly good at responding: https://github.com/moj-analytical-services/splink/discussion...
As I pointed out in 2004[3], one technique around Naive Bayes (and other machine learning systems) was to pit machine learning against machine learning[4].
[2] https://en.wikipedia.org/wiki/POPFile
[3] https://blog.jgc.org/2023/07/how-to-beat-adaptivebayesian-sp...
[4] https://en.wikipedia.org/wiki/Adversarial_machine_learning
[1] https://orange3.readthedocs.io/projects/orange-visual-progra...
Naive Bayes is also surprisingly powerful for deduplication of large datasets of messy data (and the related problem of record linkage). It's still competitive with cutting edge approaches if carefully applied, and usually much faster so it can be applied to big data. It's the engine behind Splink, the free Python library that I develop: https://github.com/moj-analytical-services/splink
Naive Bayes is also surprisingly powerful for deduplication of large datasets of messy data (and the related problem of record linkage). It's still competitive with cutting edge approaches if carefully applied, and usually much faster so it can be applied to big data. It's the engine behind Splink, the free Python library that I develop: https://github.com/moj-analytical-services/splink
I had a friend that had a record linkage problem which interested me and so I dove down a rabbit-hole of Markov logic, but didn't see any mention of Naive Bayes approach when doing so. Although I don't really have any expertise in the area anyway, so it would be easier for me to miss.
IMHO one of the most overlooked features is that it's the best eXplainable AI out there IMHO. Spam scores are easily understandable and correctable. You can even build quite digestible nomograms [1] . Still everyone is using decision trees as XAI example while in my experience they are very unstable in quite indigestible after a depth of 3. I also believe Naïve Bayes is quite undervalued. What many people however ign…
IMO logistic regression, or better fasttext (rank-limited logistic regression) should be the "default" baseline models you should use for NLP classification. They are trained with cross-entropy loss, so the scores are generally fairly well calibrated to an actual confidence score. Moreover since everything is linear (or at least logit-linear), you can do all the same explainability tricks as with NB (a little more effort for fasttext, but it is doable).