How Spam Filtering Works: From SPF to DKIM to Blacklists • Hacker News Board

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#61

Post by loup-vaillant » Sat, Sep 08, 2018, 10:53 AM UTC

walrus01 wrote: post #32

loup-vaillant wrote: post #26

Earlier quoted context omitted.

Dynamic IP is a bummer, but outright closing ports is a direct violation of net neutrality. It's baffling that it's tolerated at all. An ISP's job is to forward IP packets. Not read them. TCP/UDP ports number are the content of those IP packets—not the meta data. Filtering based on such content is already a form of discrimination. What's next, deep packet inspection? Some ISP filter by default, but they do this at th…

Counterpoint: Unless you've been in network engineering for an ISP with tens of thousands or more residential customers, you have no idea how much spam/abuse/trojaned PC traffic comes from residential users. There's a very good reason why ISPs block some of this, for the protection of their own users. It's even necessary to block port 139 windows file sharing or people will connect PCs with totally open file shares s…

Counterpoint of what exactly?

Blocking ports by default is okay, and will address the clueless user problem (clueless users won't change the defaults to begin with). It's blocking them no matter what that's evil.

Now I live in a country where most ISPs lend you a "box", a router, which is capable of doing all the blocking they want, and is configurable by the user. One of them uses that box to block the SMTP port by default, and let the users who want to change that setting.

Problem solved.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#62

Post by bradknowles » Sun, Sep 09, 2018, 8:20 PM UTC

znpy wrote: post #43

walrus01 wrote: post #32

Earlier quoted context omitted.

Counterpoint: Unless you've been in network engineering for an ISP with tens of thousands or more residential customers, you have no idea how much spam/abuse/trojaned PC traffic comes from residential users. There's a very good reason why ISPs block some of this, for the protection of their own users. It's even necessary to block port 139 windows file sharing or people will connect PCs with totally open file shares s…

> The average HN user is in like the 99.9th percentile of residential internet users. And we have the right of doing this. Anyhow: the dynamic ip problem might be solved with ipv6. The spam problem... Imho graylisting + spf + dkim + reverse-ptr should be enough to kill spamlists once and for all.

Even when gr[ae]ylisting was introduced (see " rel="nofollow">http://projects.puremagic.com/greylisting/whitepaper.html>), it was quickly recognized that there are some important sites out there that don't follow the RFCs, and therefore they break gr[ae]yisting. The page at " rel="nofollow">https://www.greylisting.org/whitelisting.php> links to a source at " rel="nofollow">http://projects.puremagic.com/web-svn/wsvn/greylisting/trunk... that doesn't even exist anymore.

For postfix, we now have the project at " rel="nofollow">https://github.com/stevejenkins/postwhite> to help automate the maintenance of whitelists of sites that break gr[ae]ylising.

IMO, we passed the point years ago when gr[ae]ylisting was still a useful technique. For anything within the last decade or so, I would say it probably does more harm than good.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#63

Post by bradknowles » Sun, Sep 09, 2018, 8:28 PM UTC

znpy wrote: post #43

walrus01 wrote: post #32

Earlier quoted context omitted.

Counterpoint: Unless you've been in network engineering for an ISP with tens of thousands or more residential customers, you have no idea how much spam/abuse/trojaned PC traffic comes from residential users. There's a very good reason why ISPs block some of this, for the protection of their own users. It's even necessary to block port 139 windows file sharing or people will connect PCs with totally open file shares s…

> The average HN user is in like the 99.9th percentile of residential internet users. And we have the right of doing this. Anyhow: the dynamic ip problem might be solved with ipv6. The spam problem... Imho graylisting + spf + dkim + reverse-ptr should be enough to kill spamlists once and for all.

See also the "Disadvantages" section at " rel="nofollow">https://en.wikipedia.org/wiki/Greylisting>, and read section 4 of RFC6647 at " rel="nofollow">https://tools.ietf.org/html/rfc6647#section-4>. Note that his RFC was written in 2012.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#64

Post by massaman_yams » Mon, Sep 10, 2018, 5:05 PM UTC

mehrdadn wrote: post #48

massaman_yams wrote: post #39

Earlier quoted context omitted.

In short: ARC builds on top of DMARC to fix some cases that DMARC didn't handle well. (And DMARC, in turn, builds on top of DKIM and SPF.) DMARC is overall a very good thing, and has significantly cut back on the effectiveness of forged email for spam/phishing, at the expense of mailing list/forwarding headaches.

Hopefully it's been good for other people... I haven't seen any benefit personally. :\ On a related note: do you know if DMARC feedback leaks information to the sender on whether or not you opened an email, or reported it as spam, or similar? If so, do you know why this isn't considered a privacy or security issue?

Recipient activity is not part of DMARC, so there are no concerns there.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#65

Post by massaman_yams » Mon, Sep 10, 2018, 5:42 PM UTC

lxchase wrote: post #3

Would love to get feedback from the HN community. We're a decently sized sender following what we think are best practices (sending to engaged, dkim, etc.) however our gmail deliverability is rock bottom and it's been difficult to improve. Every other provider is to benchmark or better. One issue may be high hard bounce rates on our very first email sent, but we don't send emails to bounces at all afterwards. (Wouldn…

Make sure SPF, DKIM, and From domains are all at the same root/parent domain. AKA "alignment" in DMARC - this can help even without a full DMARC record.

Tighten your engagement segmentation for Gmail. Repeat every 2 weeks if you don't see improvement. If you get down to 30 day actives with no improvement, the problem is almost certainly elsewhere. Occasionally it can be related to external factors - 3rd party email traffic which might use your domain, say if you buy advertising in other newsletters, bad SEO practices (think 'blackhat' rather than 'mediocre'), or poor customer experience with your business/website.

Hard bounce rates on your initial email are always a little elevated vs known recipients, but if you're above 2-3% that's too high and you should work to fix problems with the signup process.