Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Autospam and Naive Bayes

pixelfed.blog

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Autospam and Naive Bayes

#31

Post by fishywang »

jgrahamc wrote: post #16

I have some history with Naive Bayes for email filtering[1][2] and it doesn't surprise me that it's still a viable technique. It's incredibly fast to execute and for spam it was fascinating to see that as spammers tried to evade Naive Bayes they were actually increasing the numbers of tokens that made their spam look like spam. And it generalized to multiple categories. To this day I receive messages from people sayi…

Had been using popfile for years before gmail and it had always been great.

I think that's also my earliest encounter with the conception of "local web ui", while everything else at the time needs some this for configurations. It's also truly cross platform as it's written in perl.

Re: Autospam and Naive Bayes

#32

Post by jrnichols »

INTPenis wrote: post #8

I predicted this more than 2 years ago when I was running a fedi node. Back in 2004 I used to build spam filtering proxies using bayesian filters in spamassassin. And my experience in the fedi instantly made me realize they would be a quick and easy solution for a lot of fedi spam. So it's fun to see pixelfed taking the leap. I can't wait to see how it evolves. The fedi being very similar to e-mail is most likely goi…

I'm still a happy SpamAssassin user (self hosting email still) and this is the first thing I thought of as well. especially as I more often consider spinning up a pixelfed instance.

The URIBL addition really helped cut down junk, I wonder if it would help here too.

Re: Autospam and Naive Bayes

#33

Post by tensor »

sweezyjeezy wrote: post #28
tensor wrote: post #26

Earlier quoted context omitted.

Theoretically, any discriminator that maximizes the expected value rather than the underlying distribution will be better. Logistic regression is an example of a discriminative classifier. Even better, however, is if you maximize the margin, so a large margin discriminative classifier should be your baseline. I believe that the classification layer in fasttext is equivalent to logistic regression (but not large-margi…

All bag-of-words models use some kind of word vector - for a normal logistic regression / naive bayes, say "spam" and "spammer" are first and second indexed words in the vectorizer, then their word vectors are like [1, 0, 0, ....] and [0, 1, 0, ....] (length = vocab size). For both logistic regression and fasttext you get the 'document vector' by adding up their respective word vectors before applying a final linear…

Yes when I said word vectors I meant rich embeddings not one hot representations.

That said, in reviewing the fastext bag of tricks paper on their classification module I’m now second guessing my assumption that they use complex embeddings. Their architecture is otherwise exactly reproducible in vowpalwabbit, and in fact in the paper they claim that it is equivalent to a specific combination of vowpalwabbit flags.

In particular the vowpalwabbit neural network flag is needed. However, vowpalwabbit only uses one hot vectors for their ngram features. The neural network flag just adds a hidden layer.

I had assumed that fasttext uses rich word embeddings in its classifier because it has another module to train them.

If it is actually the same as vowpalwabbit, then I can say that I’ve never had the extra hidden layer really help, though as they note it does make vowpalwabbit quite slow.

Re: Autospam and Naive Bayes

#34

Post by sweezyjeezy »

tensor wrote: post #33
sweezyjeezy wrote: post #28

Earlier quoted context omitted.

All bag-of-words models use some kind of word vector - for a normal logistic regression / naive bayes, say "spam" and "spammer" are first and second indexed words in the vectorizer, then their word vectors are like [1, 0, 0, ....] and [0, 1, 0, ....] (length = vocab size). For both logistic regression and fasttext you get the 'document vector' by adding up their respective word vectors before applying a final linear…

Yes when I said word vectors I meant rich embeddings not one hot representations. That said, in reviewing the fastext bag of tricks paper on their classification module I’m now second guessing my assumption that they use complex embeddings. Their architecture is otherwise exactly reproducible in vowpalwabbit, and in fact in the paper they claim that it is equivalent to a specific combination of vowpalwabbit flags. In…

fasttext word embedding is equivalent to adding a hidden layer, as long as you DON'T put a nonlinearity on it if that helps.

Re: Autospam and Naive Bayes

#35

Post by jgrahamc »

fishywang wrote: post #31
jgrahamc wrote: post #16

I have some history with Naive Bayes for email filtering[1][2] and it doesn't surprise me that it's still a viable technique. It's incredibly fast to execute and for spam it was fascinating to see that as spammers tried to evade Naive Bayes they were actually increasing the numbers of tokens that made their spam look like spam. And it generalized to multiple categories. To this day I receive messages from people sayi…

Had been using popfile for years before gmail and it had always been great. I think that's also my earliest encounter with the conception of "local web ui", while everything else at the time needs some this for configurations. It's also truly cross platform as it's written in perl.

I remember someone saying to me in about 2005 that the "true genius" of the UI was that it had a localhost web UI. I did that because I wanted it to be cross-platform and I didn't want to use Tk or something. It just seemed natural that apss would use the web browser for configuration. And yeah, I used Perl and me (and others) worked hard to make sure it coped with all sorts of platforms.

Re: Autospam and Naive Bayes

#36

Post by rkwz »

RobinL wrote: post #19
reuben364 wrote: post #18

Earlier quoted context omitted.

I had a friend that had a record linkage problem which interested me and so I dove down a rabbit-hole of Markov logic, but didn't see any mention of Naive Bayes approach when doing so. Although I don't really have any expertise in the area anyway, so it would be easier for me to miss.

The model is usually called the Fellegi Sunter model, but once you get into the maths, it's actually the same as Naïve Bayes. If you're interested I've got a blog post that explains this here: https://www.robinlinacre.com/maths_of_fellegi_sunter/

Thanks for this, been meaning to read up on the Fellegi Sunter paper for a long time!

Re: Autospam and Naive Bayes

#37

Post by rkwz »

RobinL wrote: post #3

Naive Bayes is also surprisingly powerful for deduplication of large datasets of messy data (and the related problem of record linkage). It's still competitive with cutting edge approaches if carefully applied, and usually much faster so it can be applied to big data. It's the engine behind Splink, the free Python library that I develop: https://github.com/moj-analytical-services/splink

> Naive Bayes is also surprisingly powerful for deduplication of large datasets

Thanks for this! I implemented Entity Resolution from scratch 2 years back and was mostly able to do deterministic linkages using identifiers or using TF-IDF on names as a suggestion for human intervention. Will be writing about my learnings in upcoming months [1].

Was mostly looking at research papers before but will take a look at your project - your documentation looks good!

[1] https://www.sheshbabu.com/posts/entity-resolution-challenges...

Re: Autospam and Naive Bayes

#38

Post by pixl97 »

jgrahamc wrote: post #16

I have some history with Naive Bayes for email filtering[1][2] and it doesn't surprise me that it's still a viable technique. It's incredibly fast to execute and for spam it was fascinating to see that as spammers tried to evade Naive Bayes they were actually increasing the numbers of tokens that made their spam look like spam. And it generalized to multiple categories. To this day I receive messages from people sayi…

Yea, I remember using Bayes filtering back in the early 2000s and it was very effective. Pretty sure it was in the SpamAssassin perl application.

One thing we saw that was more problematic is spams getting shorter and the use of image only spam. Turns out a lot of human to human communications is just short single sentence blurbs between people. We saw a lot of junk spam that didn't really have a call for action, but seemed to exist just for the purpose of making Bayes classification more difficult and error prone.

Re: Autospam and Naive Bayes

#39

Post by fishywang »

fishywang wrote: post #31
jgrahamc wrote: post #16

I have some history with Naive Bayes for email filtering[1][2] and it doesn't surprise me that it's still a viable technique. It's incredibly fast to execute and for spam it was fascinating to see that as spammers tried to evade Naive Bayes they were actually increasing the numbers of tokens that made their spam look like spam. And it generalized to multiple categories. To this day I receive messages from people sayi…

Had been using popfile for years before gmail and it had always been great. I think that's also my earliest encounter with the conception of "local web ui", while everything else at the time needs some this for configurations. It's also truly cross platform as it's written in perl.

s/this/gui/, sometimes gboard makes bizarre auto corrections.
Post reply on HN