Live data from Hacker News

How Spam Filtering Works: From SPF to DKIM to Blacklists

deliciousbrains.com

41–50 of 65 posts

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#41
post #11

I've been thinking for a while that there should be a movement to allow people running mailservers at their home again. "Again" as in get other actors in the field to allow/ease that: spamlist managers should stop blocking residential address by default and make it easier to appeal. ISPs should make it possible and easy to get a reverse-ptr with every fixed ipv4 allocation. In this monitoring age I want to be able to…

It seems like there might be an opportunity here, for a hybrid service comprised of "client" software installed on your own hardware, which actually hosts and runs the email inbox, and central server-hosted software that takes care of configuring the client and all the associated authentication schemes. The server-hosted software would never see the actual emails; it would just manage the provision of email service.…

> The server-hosted software would never see the actual emails; it would just manage the provision of email service

...

> If the service was responsibly run, it could even "vouch" somehow for the email inboxes it provisions. So even though the emails come from, like, Comcast ISP address space, other ISPs would recognize a trustworthy source of email. (Dynamic IP provision would complicate this part.)

If you are willing to to let the server-hosted software have some visibility into your mail traffic (but not the content), there's a way to handle this that would not have problems with dynamic IP addresses [1].

Briefly, whenever your email client needs to send a mail through your Comcast outgoing SMTP service, it would first contact the vouching service and provide a hash of the message ID of the message it is about to send (and maybe the from address?).

The vouching service would verify that it is talking to one of its legitimate customers, and then return a line for the client to put in the header of the outgoing mail. This line would contain a copy of the message ID hash (and from address?), a timestamp, the client IP address, and would be signed using a key of the vouching service.

The client puts that line in the header and sends the mail through Comcast.

The SMTP server that Comcast relays it to can look at that line, check the signature, and if it verifies it can compare the information in that line to the corresponding information in or derived from the rest of the header. If the information all matches and the timestamp is in the recent past, accept that the mail really came from the person it claims to be from.

This would give the vouching service get a good idea of how much email you send and when you send it, and also give them hashes of your message IDs, so it is not ideal.

A problem with this is this it depends on every SMTP service that Comcast might relay your mail to (such as the SMTP server for every domain you send mail to) being set up to verify the signature of the vouching service you use. This would be a pain in the ass.

This could be addressed by having the verification of the signature and header data done at your ISP's (e.g, Comcast's) SMTP service. That way you just have to pick a vouching service that Comcast recognizes.

Comcast would then do the check on your outgoing mail to see that you have used a vouching service they recognize and the data matches. They can then add a header that says they believe the mail is verified. For any mail that they cannot verify thusly, they can add a header that says it is not verified.

Now instead of having to know about and trust your vouching service, the SMTP servers of the domains you mail to just have to trust Comcast. That seems much more likely to come about.

If Comcast (and other residential ISPs) were willing to do this, why bother with third party vouching services? Why not handle the whole thing themselves?

One reason would be that providing vouching service requires getting more involved with the details of the customer's setup than an ISP might want to deal with. They might not want to have to deal with support questions dealing with how to get the vouching plugin or whatever working with the various email clients and operating systems customers use.

[1] Maybe...I've not sat down and tried to actually design the protocols for this. This is just off the cuff.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#42
post #25
post #11

I've been thinking for a while that there should be a movement to allow people running mailservers at their home again. "Again" as in get other actors in the field to allow/ease that: spamlist managers should stop blocking residential address by default and make it easier to appeal. ISPs should make it possible and easy to get a reverse-ptr with every fixed ipv4 allocation. In this monitoring age I want to be able to…

I'd absolutely be all for it - I've been running my mail server since the 90s, and used to run it out of my closet. There are two major, somewhat interlocking problems, though: home users with compromised machines currently represent the population of home "mail servers" (spam malware) at the moment. Selling this involves convincing mail administrators that not simply blackholing all of what is currently a cesspool i…

> The second problem is that fixed-IPs, open well-known ports and DNS have become a product differentiator for ISPs. Those are "business" features; dumb ole' consumers don't "need" them.

I understand the fixed ip thing, but closing well-known ports by default should be considered a form of censorship.

It is like selling a mobile phone, except the user find out that they can only listen but cannot talk.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#43

Earlier quoted context omitted.

Dynamic IP is a bummer, but outright closing ports is a direct violation of net neutrality. It's baffling that it's tolerated at all. An ISP's job is to forward IP packets. Not read them. TCP/UDP ports number are the content of those IP packets—not the meta data. Filtering based on such content is already a form of discrimination. What's next, deep packet inspection? Some ISP filter by default, but they do this at th…

Counterpoint: Unless you've been in network engineering for an ISP with tens of thousands or more residential customers, you have no idea how much spam/abuse/trojaned PC traffic comes from residential users. There's a very good reason why ISPs block some of this, for the protection of their own users. It's even necessary to block port 139 windows file sharing or people will connect PCs with totally open file shares s…

> The average HN user is in like the 99.9th percentile of residential internet users.

And we have the right of doing this.

Anyhow: the dynamic ip problem might be solved with ipv6. The spam problem... Imho graylisting + spf + dkim + reverse-ptr should be enough to kill spamlists once and for all.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#44
post #43

Earlier quoted context omitted.

Counterpoint: Unless you've been in network engineering for an ISP with tens of thousands or more residential customers, you have no idea how much spam/abuse/trojaned PC traffic comes from residential users. There's a very good reason why ISPs block some of this, for the protection of their own users. It's even necessary to block port 139 windows file sharing or people will connect PCs with totally open file shares s…

> The average HN user is in like the 99.9th percentile of residential internet users. And we have the right of doing this. Anyhow: the dynamic ip problem might be solved with ipv6. The spam problem... Imho graylisting + spf + dkim + reverse-ptr should be enough to kill spamlists once and for all.

Good luck getting the average residential ISP to edit rDNS records for you. It's a special thing they will only do in certain circumstances and as a special one-off thing. Automated provisioning tools and customer service are not set up for it. I'm not saying that is a good thing, or satisfactory, but that's the current status quo.

I'm all in favor of ipv6 everywhere but the current reality is that a huge number of organizations presently have v4 only smtpd in their MX records, if you try to do a pure v6 approach you will break delivery to/from something like 65% of the internet.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#45
post #25
post #11

I've been thinking for a while that there should be a movement to allow people running mailservers at their home again. "Again" as in get other actors in the field to allow/ease that: spamlist managers should stop blocking residential address by default and make it easier to appeal. ISPs should make it possible and easy to get a reverse-ptr with every fixed ipv4 allocation. In this monitoring age I want to be able to…

I'd absolutely be all for it - I've been running my mail server since the 90s, and used to run it out of my closet. There are two major, somewhat interlocking problems, though: home users with compromised machines currently represent the population of home "mail servers" (spam malware) at the moment. Selling this involves convincing mail administrators that not simply blackholing all of what is currently a cesspool i…

> If just you want a mail server now, your best bet is a cheap VM somewhere.

How do you make that work? In my experience, the mail will just be blocked because the IP has no reputation and the range is probably grey/blacklisted.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#46
post #11

I've been thinking for a while that there should be a movement to allow people running mailservers at their home again. "Again" as in get other actors in the field to allow/ease that: spamlist managers should stop blocking residential address by default and make it easier to appeal. ISPs should make it possible and easy to get a reverse-ptr with every fixed ipv4 allocation. In this monitoring age I want to be able to…

IMO you still can if you have a static IP. As far as I know, dynamic IP addresses are one of the strongest filters for a lot of spam.

Have a static IP and properly setup your SPF, DKIM and DMARC...you should be good to go.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#47
post #11

I've been thinking for a while that there should be a movement to allow people running mailservers at their home again. "Again" as in get other actors in the field to allow/ease that: spamlist managers should stop blocking residential address by default and make it easier to appeal. ISPs should make it possible and easy to get a reverse-ptr with every fixed ipv4 allocation. In this monitoring age I want to be able to…

You would need to convince your ISP to allow you to modify your PTR. There are many servers that implement one form or another of the S25R methodology [1]. That is, simple regex against your FCrDNS to see if you are a generic network device or generic home user. The page I am linking to is an example of the concept, but many anti-spam tools implement this in varying degrees.

[1] - http://www.gabacho-net.jp/en/anti-spam/anti-spam-system.html

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#48

Earlier quoted context omitted.

The new hotness is ARC (arc-spec.org), which I understand came out of DMARC? Not sure though, don't know too much about it. For DMARC, it's not so awesome :\ https://news.ycombinator.com/item?id=17900765

In short: ARC builds on top of DMARC to fix some cases that DMARC didn't handle well. (And DMARC, in turn, builds on top of DKIM and SPF.) DMARC is overall a very good thing, and has significantly cut back on the effectiveness of forged email for spam/phishing, at the expense of mailing list/forwarding headaches.

Hopefully it's been good for other people... I haven't seen any benefit personally. :\

On a related note: do you know if DMARC feedback leaks information to the sender on whether or not you opened an email, or reported it as spam, or similar? If so, do you know why this isn't considered a privacy or security issue?

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#49
post #11

I've been thinking for a while that there should be a movement to allow people running mailservers at their home again. "Again" as in get other actors in the field to allow/ease that: spamlist managers should stop blocking residential address by default and make it easier to appeal. ISPs should make it possible and easy to get a reverse-ptr with every fixed ipv4 allocation. In this monitoring age I want to be able to…

I'm curious as to what privacy you think you'd gain from storing your mail at home? Unless everyone encrypts everything they send you, most likely most of that mail will go through a government monitored connection at some point. It wouldn't be that hard for them to reconstruct your mailbox if they wanted to.

Also, why not keep it on a hosted server somewhere? You could encrypt it at rest to prevent anyone coming in and getting the data, which would make it as private as storing it at home. And then you don't have to worry about deliverability or backup power as a nice side bonus.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#50
I've spent many, many hours playing with my mailserver and adding all of these extra features and security settings to try to keep Gmail and others from denying mail from me.

It turns out that no matter what I do, I'm pretty much out of luck because I'm sending from a non-standard (.link) domain which gets treated as spam by Gmail as well as others. I get a perfect score on every email test site I've tried, but I still go to spam (or worse) regularly which is a big problem if I ever want to use my personal email for anything important.

For anyone looking to host their own email using a personal domain, I'd suggest sticking to something like .com to avoid the hell that I've been through with this.

Post reply on HN