Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Spam Filtering Works: From SPF to DKIM to Blacklists

deliciousbrains.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#31

Post by walrus01 »

znpy wrote: post #11

I've been thinking for a while that there should be a movement to allow people running mailservers at their home again. "Again" as in get other actors in the field to allow/ease that: spamlist managers should stop blocking residential address by default and make it easier to appeal. ISPs should make it possible and easy to get a reverse-ptr with every fixed ipv4 allocation. In this monitoring age I want to be able to…

> I've been thinking for a while that there should be a movement to allow people running mailservers at their home again.

This will be really hard, because unless you're willing to give home users actual permanently assigned static IP addresses and treat them like medium sized businesses (IP space SWIP to customer, etc), any given /27 to /23 sized v4 DHCP pool of residential customers is going to have a ton of abuse coming from clueless people running infected routers, virus/worm/trojan infected home PCs, etc on it.

Getting the average residential ISP to do proper rDNS for your IP address is impossible in many places.

Therefore when recipient SMTPd are scoring the IP space reputation of your residential netblock, your home mail server is going to be mixed in with all the shit.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#32

Post by walrus01 »

loup-vaillant wrote: post #26
_jal wrote: post #25

Earlier quoted context omitted.

I'd absolutely be all for it - I've been running my mail server since the 90s, and used to run it out of my closet. There are two major, somewhat interlocking problems, though: home users with compromised machines currently represent the population of home "mail servers" (spam malware) at the moment. Selling this involves convincing mail administrators that not simply blackholing all of what is currently a cesspool i…

Dynamic IP is a bummer, but outright closing ports is a direct violation of net neutrality. It's baffling that it's tolerated at all. An ISP's job is to forward IP packets. Not read them. TCP/UDP ports number are the content of those IP packets—not the meta data. Filtering based on such content is already a form of discrimination. What's next, deep packet inspection? Some ISP filter by default, but they do this at th…

Counterpoint: Unless you've been in network engineering for an ISP with tens of thousands or more residential customers, you have no idea how much spam/abuse/trojaned PC traffic comes from residential users. There's a very good reason why ISPs block some of this, for the protection of their own users. It's even necessary to block port 139 windows file sharing or people will connect PCs with totally open file shares straight into their cable or DSL modems.

It's a real headache. The average HN user is in like the 99.9th percentile of residential internet users. HN users, give them a proper DIA feed, sure. Everyone else, I'm wary.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#33

Post by walrus01 »

_urga wrote: post #5

Much more important than SPF and DKIM is Forward-confirmed reverse DNS. You need SPF and DKIM, but before that you need FCrDNS. Without FCrDNS, your server will look like a dynamic IP address. It's also not enough to monitor your own dedicated IP address. You need to monitor your entire /24 IP address neighborhood at your hosting company. You might be sharing the same /24 IP address space as a hacked Wordpress instal…

rDNS is of course extremely important. That was important even long before the existence of SPF and DKIM.

The neighboring IP space and the general IP space reputation of the ARIN/RIPE/APNIC/AFRINIC (or whatever) block assigned to your ISP is extremely important. Your neighbours can ruin everything for you. The only way to ensure that this doesn't become an issue is to host your smtpd at an ISP that has a highly attentive, clueful NOC and responsive abuse policy that actually deals with network abuse (real abuse, not DMCA notices).

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#34

Post by walrus01 »

Sir_Cmpwn wrote: post #23

If you're looking for an easy way to test your outgoing mail setup, I recommend mail-tester.com. They give you an address to send an email to, then analyse it and tell you how you can make it more deliverable.

https://mxtoolbox.com/ can also give you a lot of useful info related to your MX.

https://mxtoolbox.com/diagnostic.aspx

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#35

Post by megous »

_jal wrote: post #25
znpy wrote: post #11

I've been thinking for a while that there should be a movement to allow people running mailservers at their home again. "Again" as in get other actors in the field to allow/ease that: spamlist managers should stop blocking residential address by default and make it easier to appeal. ISPs should make it possible and easy to get a reverse-ptr with every fixed ipv4 allocation. In this monitoring age I want to be able to…

I'd absolutely be all for it - I've been running my mail server since the 90s, and used to run it out of my closet. There are two major, somewhat interlocking problems, though: home users with compromised machines currently represent the population of home "mail servers" (spam malware) at the moment. Selling this involves convincing mail administrators that not simply blackholing all of what is currently a cesspool i…

Other way I've thought of effectively achieving this is by using VPS for static IP outside of residential ranges and home machine connected by WireGuard VPN. WireGuard handles roaming easilly, so dynamic IP is no problem, IPv6 is no problem, and the mail will stay on your home machine. You'll also bypass your country's/ISP tracking somewhat if you place your VPS outside of the country.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#36

Post by walrus01 »

lxchase wrote: post #3

Would love to get feedback from the HN community. We're a decently sized sender following what we think are best practices (sending to engaged, dkim, etc.) however our gmail deliverability is rock bottom and it's been difficult to improve. Every other provider is to benchmark or better. One issue may be high hard bounce rates on our very first email sent, but we don't send emails to bounces at all afterwards. (Wouldn…

a) SPF correct?

b) DKIM correct and signing properly?

c) rDNS is right?

c) the IP block your smtpd is in isn't in any RBLs?

d) You're not sending commercial bulk email?

e) Your mail server identifies itself properly to the other smtpd?

f) Your mail server properly implements TLS1.2 opportunistically?

I run my own mail server and have zero delivery issues to gmail. But I do not run marketing mailing lists, nor bulk communications of the same mail content to many customers, it sounds like you're mass-sending mail.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#37

Post by flurdy »

znpy wrote: post #11

I've been thinking for a while that there should be a movement to allow people running mailservers at their home again. "Again" as in get other actors in the field to allow/ease that: spamlist managers should stop blocking residential address by default and make it easier to appeal. ISPs should make it possible and easy to get a reverse-ptr with every fixed ipv4 allocation. In this monitoring age I want to be able to…

I did run mailserver(s) at home for a while in early 2000s.

Proved difficult when I once went on holiday as I switched it off and then it got flooded with emails (mostly spam) when I got back (from backup mxs run by friends). Had my ISP call me as worried my PC was on a botnet due to sudden traffic. Explained I was running Linux, and they immediately said: "thats fine then have a nice day". Thought perhaps they would need more detail but no. ¯\_(ツ)_/¯

As others have said. Today's cheap nano-esque cloud instances (with swap disk enabled) on AWS and similar is easier to have running 24/7, and less risk it will burn your house down...

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#38

Post by megous »

rurcliped wrote: post #21

The great fallacy of spam filtering is that access control SHOULD be probabilistic whenever it is easy to implement that. Let's look at the physical world where implementation is harder. It's very unlikely that I'll arrive home from work at 4:30 AM, AND that I'll be driving a rental car instead of my own car, AND that I'll be wearing new shoes with a sole pattern that my smart walkway hasn't seen before. So, ideally…

I don't think that's how it works. You train your spam filter both with spam and ham and the sets need to have comparable sizes for it to work reliably.

So in your example, you'd train it on your patterns of behavior AND patterns of burglars. So unless you had a clear burglarly behavior, the system would probably not flag you out for just behaving differently.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#39

Post by massaman_yams »

mehrdadn wrote: post #22
gorkish wrote: post #9

The author is still pretty far behind the curve with this info. Unfortunately while these policies are great to have (well these days they are more or less necessary), the simple fact is that they are so often misused or improperly maintained they don't really stop that much; a SPF mismatch is simply treated as another item to score the likeliness that a message should or should not be blocked. A DKIM signature is al…

The new hotness is ARC (arc-spec.org), which I understand came out of DMARC? Not sure though, don't know too much about it. For DMARC, it's not so awesome :\ https://news.ycombinator.com/item?id=17900765

In short: ARC builds on top of DMARC to fix some cases that DMARC didn't handle well. (And DMARC, in turn, builds on top of DKIM and SPF.)

DMARC is overall a very good thing, and has significantly cut back on the effectiveness of forged email for spam/phishing, at the expense of mailing list/forwarding headaches.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#40

Post by jonathanbull »

Nice article. It's worth noting that it's not just the reputation of the sender domain you should be looking at - the reputation of the links in the body also plays a big part. In fact over at https://emailoctopus.com, this is the number one issue we see for legitimate senders landing in spam.
Post reply on HN