Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern Anti-Spam and E2E Crypto (2014)

moderncrypto.org

Post reply on HN
Print view View original

31–40 of 49 posts

Re: Modern Anti-Spam and E2E Crypto (2014)

#31

Post by dochtman »

tempestn wrote: post #26
dochtman wrote: post #22

Earlier quoted context omitted.

I had similar problems. On top of that, I forward my email through my own server to GMail (so I control the domain, but can use the GMail ecosystem as UX), and this was posing problems because GMail would greylist my server quite a bit for sending in too much spam. I now run rspamd on my own server, which does a pretty great job. With properly training the bayes filters it has, I now receive on the order of 3 spam me…

Actually, that's the exact situation I'm in as well. So to clarify, the spam that gets through rspamd lands in gmail's spam folder, so you still need to manually check that, but all the obviously-spam stuff has been cut out before it ever got to gmail (solving both your problems). Sounds like exactly what I asked for! Interesting your mention about being grey-listed too. How did you determine that happened? Presumabl…

rspamd has three levels of handling, depending on the spam score: (1) ham, which gets passed through, (2) spam, which does not, and (3) "not sure", which gets passed through but gets headers attached with the spam score and how the score is built up. So I get (1) and (3) in my GMail account, but all the stuff for which rspamd is confident it's spam no longer makes it into GMail.

Of course, rspamd lets you tweak the thresholds for these levels. For example, after a while I lowered the threshold for "spam", increasing the amount of stuff that gets discarded by rspamd, because I noticed that rspamd was doing a pretty good job of scoring, and the false positives I was seeing had a lower score anyway.

I'm actually not sure grey-listing is the correct term, but I noticed in my server's MTA log that Google was rate limiting me a lot because my server was sending through significant amounts of email. This was also noticeable sometimes because it would take quite some time for email to get through, which I found annoying.

Yes, you probably want to run SRS as well, otherwise GMail will be unable to correctly understand your headers. However, this effectively puts your server on the hook for any email forwarded; this is why I don't think you want to go there without also putting some kind of spam filter in place, otherwise I assume your server's reputation will deteriorate.

Re: Modern Anti-Spam and E2E Crypto (2014)

#32

Post by ikeboy »

tempestn wrote: post #15
haimez wrote: post #14

Earlier quoted context omitted.

maybe, but you have to start with a better definition of spam. Currently, you're describing it as "something that you wouldn't want to read" where even a fellow human might not hit the mark 100% of the time. Machine learning requires large data sets and training, and mushy targets like your inbox are tricky because it's hard to tell a computer what its score was on a given attempt- even with human scoring.

To clarify, I'm not expecting something to identify the exact messages I'd want to read. But if it could take the 3000 messages in my spam folder, and separate out 500 "probably spam" from 2500 "definitely spam", it would cut my manual spam-scanning time down significantly.

Can you identify the top 5-10 keywords in spam that never appear in legitimate emails, and set up a filter to discard those immediately or move them somewhere else?

Re: Modern Anti-Spam and E2E Crypto (2014)

#33

Post by fiatjaf »

Spam is a problem everywhere except Gmail. People have an illusion that the battle against spam is won because Gmail did the job. All the other email providers struggle everyday. SpamAssassin is worthless, it is a piece of shit that does nothing.

We desperately need a service that helps filtering email -- like, for example, something simple that just accepts reports about some email being spam or not, and creates a list of spam addresses.

Re: Modern Anti-Spam and E2E Crypto (2014)

#35

Post by jimktrains2 »

fiatjaf wrote: post #33

Spam is a problem everywhere except Gmail. People have an illusion that the battle against spam is won because Gmail did the job. All the other email providers struggle everyday. SpamAssassin is worthless, it is a piece of shit that does nothing. We desperately need a service that helps filtering email -- like, for example, something simple that just accepts reports about some email being spam or not, and creates a l…

I have to check my Spam folder for false positive constantly, so it's not as if GMail has solved the SPAM problem.

Re: Modern Anti-Spam and E2E Crypto (2014)

#36

Post by jimktrains2 »

> Botnets appeared as a way to get around RBLs, and in response spam fighters mapped out the internet to create a "policy block list" - ranges of IPs that were assigned to residential connections and thus should not be sending any email at all.

While I understand that _some_ residential ISPs don't let you run services on your connection, policies like this make me sad because it means the web is becoming more-and-more something you need other people to do for you.

Re: Modern Anti-Spam and E2E Crypto (2014)

#37

Post by pmlnr »

fiatjaf wrote: post #33

Spam is a problem everywhere except Gmail. People have an illusion that the battle against spam is won because Gmail did the job. All the other email providers struggle everyday. SpamAssassin is worthless, it is a piece of shit that does nothing. We desperately need a service that helps filtering email -- like, for example, something simple that just accepts reports about some email being spam or not, and creates a l…

> SpamAssassin is worthless, it is a piece of shit that does nothing.

This is not entirely true. Spamassassin, dspam, and all the bayesian based ones need constrant training and feedback loop to work. The time trainings lasts for is getting shorter and shorter, but it's not entirely inefficient. ( I'm running dspam on my mail box. )

Combine this with weighted blacklists ( postscreen in my case ), add dkim and dmarc checks and it's fine for a small provider. Far from ideal, but working.

I'd love to see an open source implementation of what google was referring to as domain based trust, but it's also a nasty thing.

I've recently tried to change my mail address from a .eu domain to a .net, and most of my mail landed in the recipients' spam folder. It's a fresh domain, no one ever sent anything from it, so I'd assume fresh domains are untrusted by default, which is really bad and is generally wrong. If the trust is not OK by default, spam is users consider it spam, that's going to kill domain based mailing, which is horrible, and gmail will be the one to blaim when we have not alternatives to few providers.

Re: Modern Anti-Spam and E2E Crypto (2014)

#38

Post by pmlnr »

jimktrains2 wrote: post #36

> Botnets appeared as a way to get around RBLs, and in response spam fighters mapped out the internet to create a "policy block list" - ranges of IPs that were assigned to residential connections and thus should not be sending any email at all. While I understand that _some_ residential ISPs don't let you run services on your connection, policies like this make me sad because it means the web is becoming more-and-mor…

    smtpd_helo_restrictions = permit_mynetworks,
      reject_invalid_helo_hostname,
      permit

    smtpd_recipient_restrictions =  permit_mynetworks,
      permit_sasl_authenticated,
      reject_invalid_hostname,
      reject_non_fqdn_recipient,
      reject_unknown_recipient_domain,
      reject_unauth_pipelining,
      reject_unauth_destination,
      check_policy_service unix:private/policy-spf,
      check_client_access    pcre:${config_directory}/dspam_filter_access,
      permit
This is in my postfix config. The invalid hostname and non_fqdn tests are working quite well against residential hosts: they don't have valid reverse DNS or an fqdn, and so they get eliminated fast.

Re: Modern Anti-Spam and E2E Crypto (2014)

#39

Post by pmlnr »

mistermumble wrote: post #7

The content in this article about anti-spam techniques is fascinating. But the title should be edited to indicate its vintage: 2014. Also, by the by, there is some interesting context about the author (Mike Hearn) and his recent activities that are not related to the topic of spam. Warning: off-topic digression below. Mike Hearn became one of the most visible Bitcoin core developers, working in that community for 5 y…

/offtopic

> vintage: 2014.

Email is from the 70s and you label anything from 2014 related to it as vintage?

Post reply on HN