How Spam Filtering Works: From SPF to DKIM to Blacklists
deliciousbrains.com
How Spam Filtering Works: From SPF to DKIM to Blacklists
1–10 of 65 posts
Re: How Spam Filtering Works: From SPF to DKIM to Blacklists
#2In my experience, spam blacklists have significantly decreased in efficiency over the last 10 years. I think the biggest e-mail providers stopped contributing to them, so the user-reported lists are almost unused. The honeypot lists lag behind the spammers by a few days, so plenty slip through. They do trim out 85% of my incoming spam, but that last 15% is still a lot. Back in ~2013 they cut out more like 99%.
Today, the single most effective thing you can do if you run your own mail server is to completely block all gTLDs. Screw 'em, they are 99.9999% spam.
Plenty of spam has valid SPF and DKIM records. They are sent through legit services, either through cracked credentials of real users or rotating through new accounts.
It also doesn't seem like anybody cares about abuse@/spam@ reports anymore...
Re: How Spam Filtering Works: From SPF to DKIM to Blacklists
#3Would love to get feedback from the HN community. We're a decently sized sender following what we think are best practices (sending to engaged, dkim, etc.) however our gmail deliverability is rock bottom and it's been difficult to improve. Every other provider is to benchmark or better. One issue may be high hard bounce rates on our very first email sent, but we don't send emails to bounces at all afterwards. (Wouldn't a double opt-in result in the same situation). What are the most impactful levers to pull in your opinion?
Re: How Spam Filtering Works: From SPF to DKIM to Blacklists
#4In my experience, spam blacklists have significantly decreased in efficiency over the last 10 years. I think the biggest e-mail providers stopped contributing to them, so the user-reported lists are almost unused. The honeypot lists lag behind the spammers by a few days, so plenty slip through. They do trim out 85% of my incoming spam, but that last 15% is still a lot. Back in ~2013 they cut out more like 99%. Today,…
Spam filtering is no better than it was 10 years ago.
Re: How Spam Filtering Works: From SPF to DKIM to Blacklists
#5Much more important than SPF and DKIM is Forward-confirmed reverse DNS.
You need SPF and DKIM, but before that you need FCrDNS.
Without FCrDNS, your server will look like a dynamic IP address.
It's also not enough to monitor your own dedicated IP address. You need to monitor your entire /24 IP address neighborhood at your hosting company. You might be sharing the same /24 IP address space as a hacked Wordpress installation and find that Outlook.com will block you just because of that.
After that, you might run into issues with large SpamAssassin setups hosting hundreds of thousands of mailboxes blocking your email because of arbitrary rules such as too little text combined with a large image etc. Plain text email will improve your deliverability for these setups.
Re: How Spam Filtering Works: From SPF to DKIM to Blacklists
#6Would love to get feedback from the HN community. We're a decently sized sender following what we think are best practices (sending to engaged, dkim, etc.) however our gmail deliverability is rock bottom and it's been difficult to improve. Every other provider is to benchmark or better. One issue may be high hard bounce rates on our very first email sent, but we don't send emails to bounces at all afterwards. (Wouldn…
> ...our gmail deliverability is rock bottom and it's been difficult to improve. Every other provider is to benchmark or better. ... What are the most impactful levers to pull in your opinion?
I'll be blunt, and maybe others will disagree with me, and say that I don't think there are any impactful levers you can pull.
In my experience, Gmail frankly does not give a shit about receiving e-mail from any senders except ones that fall in one of two categories: a) very large so users will complain in large groups if receiving from them is impacted (put your Outlooks and your Yahoos and your huge residential ISPs in this category) or b) technically-savvy senders that aren't quite as large but Gmail employees routinely interact with so they're otherwise "trustworthy" (Fastmail is an example here).
I've ranted about here but I've had terrible success being smaller than a multinational ISP and sending e-mail to Gmail users. Messages will silently disappear, even though Gmail's SMTP server claims to accept the message, and the postmaster tools are less-than-useful (that might be my one suggestion, sign up for them on behalf of your domain and see if you get any reports; I didn't, 'too small'), and no amount of tinkering or asking would help.
I admit I'm just a SMB org whose business isn't sending e-mail--it's "just" a communications tool for us--so I had this option: I threw in the towel and switched to Fastmail and have had no problems since.
Re: How Spam Filtering Works: From SPF to DKIM to Blacklists
#7In my experience, spam blacklists have significantly decreased in efficiency over the last 10 years. I think the biggest e-mail providers stopped contributing to them, so the user-reported lists are almost unused. The honeypot lists lag behind the spammers by a few days, so plenty slip through. They do trim out 85% of my incoming spam, but that last 15% is still a lot. Back in ~2013 they cut out more like 99%. Today,…
> completely block all gTLDs. Screw 'em, they are 99.9999% spam
[citation urgently needed]
Anyway, in my own experience with many years of self-hosting mail - until giving up and going Fastmail a couple of years ago - the real problems were in sending. No matter what rigorous level of DKIM'ing and ip-hygiene and whatnot, Google and Microsoft - Microsoft to a grotesque degree - would randomly ditch incoming mails from my server. Would sometimes happen in the middle of a conversation thread, and for the most part without warning. The kind of person using Hotmail is typically not someone you can convince that the error lies in his end.
Re: How Spam Filtering Works: From SPF to DKIM to Blacklists
#8Would love to get feedback from the HN community. We're a decently sized sender following what we think are best practices (sending to engaged, dkim, etc.) however our gmail deliverability is rock bottom and it's been difficult to improve. Every other provider is to benchmark or better. One issue may be high hard bounce rates on our very first email sent, but we don't send emails to bounces at all afterwards. (Wouldn…
> ...our gmail deliverability is rock bottom and it's been difficult to improve. Every other provider is to benchmark or better. ... What are the most impactful levers to pull in your opinion? I'll be blunt, and maybe others will disagree with me, and say that I don't think there are any impactful levers you can pull. In my experience, Gmail frankly does not give a shit about receiving e-mail from any senders except…
> That might be my one suggestion, sign up for them on behalf of your domain and see if you get any reports; I didn't, 'too small.
I had the same issue, what I found though is that DMARC reports give at least some insight into the process, but that's about all the information I've managed to get from Google.
Re: How Spam Filtering Works: From SPF to DKIM to Blacklists
#9The author is still pretty far behind the curve with this info. Unfortunately while these policies are great to have (well these days they are more or less necessary), the simple fact is that they are so often misused or improperly maintained they don't really stop that much; a SPF mismatch is simply treated as another item to score the likeliness that a message should or should not be blocked. A DKIM signature is almost completely useless in this system since all it can do is prove a message was handled by a specific domain holder and not modified. So if you want to modify it, just remove the signature. If you want to send a forged message, don't sign it. There's nothing in DKIM itself to tell a recipient "my messages must be signed."
The new hotness is a DMARC record, and it finally allows mail senders to basically say "Here are my suggested handling policies for DKIM and SPF, and please actually enforce them because they are properly maintained and tested." Best of all it has a feedback loop, so you are able get reports of how receiving hosts are treating your messages and forensic reports of forgeries which can help with detection of several types of phishing threats.
It's still a long way from perfect, but it's better. It doesn't help deliverability though. For that you will want to stop emailing shit that people don't want.
Re: How Spam Filtering Works: From SPF to DKIM to Blacklists
#10Much more important than SPF and DKIM is Forward-confirmed reverse DNS. You need SPF and DKIM, but before that you need FCrDNS. Without FCrDNS, your server will look like a dynamic IP address. It's also not enough to monitor your own dedicated IP address. You need to monitor your entire /24 IP address neighborhood at your hosting company. You might be sharing the same /24 IP address space as a hacked Wordpress instal…
Yes, Microsoft blocked me sending email from my shared hosting to my Hotmail. The email address was whitelisted, and established email back and forth to that domain, and no spam ... but another server at the same host had previously been blacklisted in one blacklist.
SPF nor DKIM helped, I don't own the mail server nor want to pay Microsoft's service provider to verify me.
In the end I had to bounce the emails from a third-party address. Which you'd think shouldn't work -- certainly not when whitelisting doesn't -- but solved the problem.
So to avoid being flagged as a spammer, do the spammiest thing possible: set up a new, free mailbox and bounce your mail from there. SMH.