Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Spam Filtering Works: From SPF to DKIM to Blacklists

deliciousbrains.com

Post reply on HN
Print view View original

11–20 of 65 posts

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#11

Post by znpy »

I've been thinking for a while that there should be a movement to allow people running mailservers at their home again.

"Again" as in get other actors in the field to allow/ease that: spamlist managers should stop blocking residential address by default and make it easier to appeal. ISPs should make it possible and easy to get a reverse-ptr with every fixed ipv4 allocation.

In this monitoring age I want to be able to handle my private conversations (and my data) on my own iron (running somewhere in my house).

All this impediments to running your own mailserver and take back control of your own data really look like a push towards state-/corporate-backed espionage of individuals.

It's not 1995 anymore, most of us have an internet connection well capable of sustain enough mailboxes for a whole family.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#12

Post by teekert »

Been there, done that. And then Microsoft answers my complaints (after my mails never showed up at my brother in laws outlook.com mailbox):

We have reviewed your IP(...) and determined that messages are being filtered based on the recommendations of the SmartScreen® Filter.

Email filtering is based on many factors, but primarily it's due to mail content and recipient interaction with that mail. Because of the proprietary nature of SmartScreen® and because SmartScreen® Filter technology is always adapting and learning more about what is and isn't unwanted mail, it is not possible for us to offer specific advice about improving your mail content. However, in general SmartScreen® Filter evaluates specific words or characteristics from each e-mail message and weights them, based on their likelihood to indicate that a message is unwanted or legitimate mail.

Unfortunately, after reviewing the information you provided and in compliance with our mail policies, we are unable to offer immediate mitigation for your deliverability issue. However, we have some specific recommendations for you to consider that can help you to improve deliverability over time.

It's a struggle running your own mailserver.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#13

Post by snowwrestler »

znpy wrote: post #11

I've been thinking for a while that there should be a movement to allow people running mailservers at their home again. "Again" as in get other actors in the field to allow/ease that: spamlist managers should stop blocking residential address by default and make it easier to appeal. ISPs should make it possible and easy to get a reverse-ptr with every fixed ipv4 allocation. In this monitoring age I want to be able to…

It seems like there might be an opportunity here, for a hybrid service comprised of "client" software installed on your own hardware, which actually hosts and runs the email inbox, and central server-hosted software that takes care of configuring the client and all the associated authentication schemes.

The server-hosted software would never see the actual emails; it would just manage the provision of email service. Perhaps the server could store locally-encrypted inbox backups too.

If the service was responsibly run, it could even "vouch" somehow for the email inboxes it provisions. So even though the emails come from, like, Comcast ISP address space, other ISPs would recognize a trustworthy source of email. (Dynamic IP provision would complicate this part.)

But the big question is... what's the addressable market?? It would likely be even more expensive than something like ProtonMail. You're not going to build much of a business off of 347 neckbeard paranoics. (I joke with love.)

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#14

Post by mrmekon »

interfixus wrote: post #7
mrmekon wrote: post #2

In my experience, spam blacklists have significantly decreased in efficiency over the last 10 years. I think the biggest e-mail providers stopped contributing to them, so the user-reported lists are almost unused. The honeypot lists lag behind the spammers by a few days, so plenty slip through. They do trim out 85% of my incoming spam, but that last 15% is still a lot. Back in ~2013 they cut out more like 99%. Today,…

> completely block all gTLDs. Screw 'em, they are 99.9999% spam [citation urgently needed] Anyway, in my own experience with many years of self-hosting mail - until giving up and going Fastmail a couple of years ago - the real problems were in sending. No matter what rigorous level of DKIM'ing and ip-hygiene and whatnot, Google and Microsoft - Microsoft to a grotesque degree - would randomly ditch incoming mails from…

> completely block all gTLDs. Screw 'em, they are 99.9999% spam

That's for _my_ e-mail. Your experience may differ. Perhaps you communicate often with people on .loan domains.

E-mailing Microsoft accounts is just not an option. They have no process for fixing incorrectly blocked IPs. I take the same approach with Microsoft as with gTLDs...

Never had a problem with any other mail provider. Google has never blocked or spam-holed me.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#15

Post by snowwrestler »

lxchase wrote: post #3

Would love to get feedback from the HN community. We're a decently sized sender following what we think are best practices (sending to engaged, dkim, etc.) however our gmail deliverability is rock bottom and it's been difficult to improve. Every other provider is to benchmark or better. One issue may be high hard bounce rates on our very first email sent, but we don't send emails to bounces at all afterwards. (Wouldn…

> One issue may be high hard bounce rates on our very first email sent, but we don't send emails to bounces at all afterwards.

This is a big red flag, though. Are you saying it just happened one single time? In that case, change your IP and start over.

Or are you saying that you regularly trigger a lot of hard bounces? Because if that is the case, that might explain it all right there. High rates of hard bounces are taken as evidence of spam or other nefarious activities, since the typical reasons for high hard bounces are a) a really old list (implies it has been transacted somehow, therefore not opt-in), or b) enumeration attempts.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#16

Post by ouid »

snowwrestler wrote: post #13
znpy wrote: post #11

I've been thinking for a while that there should be a movement to allow people running mailservers at their home again. "Again" as in get other actors in the field to allow/ease that: spamlist managers should stop blocking residential address by default and make it easier to appeal. ISPs should make it possible and easy to get a reverse-ptr with every fixed ipv4 allocation. In this monitoring age I want to be able to…

It seems like there might be an opportunity here, for a hybrid service comprised of "client" software installed on your own hardware, which actually hosts and runs the email inbox, and central server-hosted software that takes care of configuring the client and all the associated authentication schemes. The server-hosted software would never see the actual emails; it would just manage the provision of email service.…

theres a lot more than 347 of those

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#17

Post by detaro »

mrmekon wrote: post #2

In my experience, spam blacklists have significantly decreased in efficiency over the last 10 years. I think the biggest e-mail providers stopped contributing to them, so the user-reported lists are almost unused. The honeypot lists lag behind the spammers by a few days, so plenty slip through. They do trim out 85% of my incoming spam, but that last 15% is still a lot. Back in ~2013 they cut out more like 99%. Today,…

I assume you mean "new gTLDs", or are you seriously blocking .com/.org/.net?

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#18

Post by nickphx »

lxchase wrote: post #3

Would love to get feedback from the HN community. We're a decently sized sender following what we think are best practices (sending to engaged, dkim, etc.) however our gmail deliverability is rock bottom and it's been difficult to improve. Every other provider is to benchmark or better. One issue may be high hard bounce rates on our very first email sent, but we don't send emails to bounces at all afterwards. (Wouldn…

Gmail is all about user engagement. If you are sending mail that users are not reading or are reacting negatively to, you will eventually be blocked. Have you used the posmaster tools they make available for delivery insight?

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#19

Post by megous »

teekert wrote: post #12

Been there, done that. And then Microsoft answers my complaints (after my mails never showed up at my brother in laws outlook.com mailbox): We have reviewed your IP( . . . ) and determined that messages are being filtered based on the recommendations of the SmartScreen® Filter. Email filtering is based on many factors, but primarily it's due to mail content and recipient interaction with that mail. Because of the pro…

Yes, it's annoying because mailbox owners have zero control too. If your brother in law complained to outlook that some emails are missing from his inbox/spam because outlook decided to reject them outright, he would get the same respnse. "We have smarter systems(r) than your judgement, sorry." (Remeber, you can't click "not spam", if the e-mail was not delivered at all)

The problem is the burden of not upsetting a mail filter got somehow shifted to the sender in case of big services filtering mail, which is ridiculous. Customers should talk to their providers if they're not receiving mail, because providers don't deliver it. But the correcting feedback loop is not there or/alternatively is much longer with more steps than in case of sender getting a bounce.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#20

Post by megous »

detaro wrote: post #17
mrmekon wrote: post #2

In my experience, spam blacklists have significantly decreased in efficiency over the last 10 years. I think the biggest e-mail providers stopped contributing to them, so the user-reported lists are almost unused. The honeypot lists lag behind the spammers by a few days, so plenty slip through. They do trim out 85% of my incoming spam, but that last 15% is still a lot. Back in ~2013 they cut out more like 99%. Today,…

I assume you mean "new gTLDs", or are you seriously blocking .com/.org/.net?

You can get far by blocking national domains like mx, ua, br, etc. If you're not expecting email from residents of said countries.
Post reply on HN