Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Spam Filtering Works: From SPF to DKIM to Blacklists

deliciousbrains.com

Post reply on HN
Print view View original

51–60 of 65 posts

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#51

Post by gorkish »

mehrdadn wrote: post #22
gorkish wrote: post #9

The author is still pretty far behind the curve with this info. Unfortunately while these policies are great to have (well these days they are more or less necessary), the simple fact is that they are so often misused or improperly maintained they don't really stop that much; a SPF mismatch is simply treated as another item to score the likeliness that a message should or should not be blocked. A DKIM signature is al…

The new hotness is ARC (arc-spec.org), which I understand came out of DMARC? Not sure though, don't know too much about it. For DMARC, it's not so awesome :\ https://news.ycombinator.com/item?id=17900765

DMARC is for the sender ARC is for the receiver/intermediary. The problem with DMARC (really it's not a problem with DMARC but we'll play like it is) is that if you do something like set up a Gmail account to forward mail to your "real" yahoo account then yahoo might not accept the message from Gmail since from yahoo's perspective Gmail is trying to dump a forged message into your box that they didn't originate. As a sender I don't really give a shit that the other end is broken by some lazy person.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#52

Post by mehrdadn »

gorkish wrote: post #51
mehrdadn wrote: post #22

Earlier quoted context omitted.

The new hotness is ARC (arc-spec.org), which I understand came out of DMARC? Not sure though, don't know too much about it. For DMARC, it's not so awesome :\ https://news.ycombinator.com/item?id=17900765

DMARC is for the sender ARC is for the receiver/intermediary. The problem with DMARC (really it's not a problem with DMARC but we'll play like it is) is that if you do something like set up a Gmail account to forward mail to your "real" yahoo account then yahoo might not accept the message from Gmail since from yahoo's perspective Gmail is trying to dump a forged message into your box that they didn't originate. As a…

How is the message considered forged when it passes DKIM though?

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#53

Post by spc476 »

mrmekon wrote: post #2

In my experience, spam blacklists have significantly decreased in efficiency over the last 10 years. I think the biggest e-mail providers stopped contributing to them, so the user-reported lists are almost unused. The honeypot lists lag behind the spammers by a few days, so plenty slip through. They do trim out 85% of my incoming spam, but that last 15% is still a lot. Back in ~2013 they cut out more like 99%. Today,…

Three years ago I investigated the effectiveness of some realtime black lists (RBL) [1]. I didn't find it worth implementing.

I also looked into the result of SPF [2] and again, found it not worth implementing. I rechecked the results for SPF earlier this year [3] and the results were the same.

[1] http://boston.conman.org/2015/05/11.1

[2] http://boston.conman.org/2015/04/12.1

[3] http://boston.conman.org/2018/01/10.1

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#54

Post by lsh123 »

megous wrote: post #35
_jal wrote: post #25

Earlier quoted context omitted.

I'd absolutely be all for it - I've been running my mail server since the 90s, and used to run it out of my closet. There are two major, somewhat interlocking problems, though: home users with compromised machines currently represent the population of home "mail servers" (spam malware) at the moment. Selling this involves convincing mail administrators that not simply blackholing all of what is currently a cesspool i…

Other way I've thought of effectively achieving this is by using VPS for static IP outside of residential ranges and home machine connected by WireGuard VPN. WireGuard handles roaming easilly, so dynamic IP is no problem, IPv6 is no problem, and the mail will stay on your home machine. You'll also bypass your country's/ISP tracking somewhat if you place your VPS outside of the country.

I run home mail sever for a while. I used to have smtp at home but a few years back I switched to a smtp proxy/forwarder on AWS ($5-6/month) connected via TLS SMTP on custom port at home (which also serves IMAPS on a custom port). Once a year I have to update DNS when my Comcast IP changes. Otherwise it is completely trouble free setup.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#55

Post by Ionlyprograminc »

mehrdadn wrote: post #52
gorkish wrote: post #51

Earlier quoted context omitted.

DMARC is for the sender ARC is for the receiver/intermediary. The problem with DMARC (really it's not a problem with DMARC but we'll play like it is) is that if you do something like set up a Gmail account to forward mail to your "real" yahoo account then yahoo might not accept the message from Gmail since from yahoo's perspective Gmail is trying to dump a forged message into your box that they didn't originate. As a…

How is the message considered forged when it passes DKIM though?

A message can still be considered a DMARC failure since the check depends on both SPF and DKIM. You can have various failures like recipient rewrites and general SPF/DKIM failures which will fail DMARC a significant amount of the time.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#56

Post by Ionlyprograminc »

mehrdadn wrote: post #48
massaman_yams wrote: post #39

Earlier quoted context omitted.

In short: ARC builds on top of DMARC to fix some cases that DMARC didn't handle well. (And DMARC, in turn, builds on top of DKIM and SPF.) DMARC is overall a very good thing, and has significantly cut back on the effectiveness of forged email for spam/phishing, at the expense of mailing list/forwarding headaches.

Hopefully it's been good for other people... I haven't seen any benefit personally. :\ On a related note: do you know if DMARC feedback leaks information to the sender on whether or not you opened an email, or reported it as spam, or similar? If so, do you know why this isn't considered a privacy or security issue?

DMARC feedback is only based on the MTA (Message Transport Agent) handling the mail. I don't have much experience with DMARC reports but Senders typically use it when they are first Setting up DMARC. It is up to the receiving MTA to send the inbox/spam result according to the RFC. I think that the reports are aggregated so single inboxes are not able to be detected. I don't think many people actually send / follow up on DMARC reports imo.

Source: https://tools.ietf.org/html/rfc7489#section-7.3

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#57

Post by canhascodez »

walrus01 wrote: post #32
loup-vaillant wrote: post #26

Earlier quoted context omitted.

Dynamic IP is a bummer, but outright closing ports is a direct violation of net neutrality. It's baffling that it's tolerated at all. An ISP's job is to forward IP packets. Not read them. TCP/UDP ports number are the content of those IP packets—not the meta data. Filtering based on such content is already a form of discrimination. What's next, deep packet inspection? Some ISP filter by default, but they do this at th…

Counterpoint: Unless you've been in network engineering for an ISP with tens of thousands or more residential customers, you have no idea how much spam/abuse/trojaned PC traffic comes from residential users. There's a very good reason why ISPs block some of this, for the protection of their own users. It's even necessary to block port 139 windows file sharing or people will connect PCs with totally open file shares s…

OT: most developers have probably not been in network engineering for an ISP. If you've written anything about your experiences, or if you could be persuaded to, I believe that a lot of people on HN would be keenly interested (myself included).

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#58

Post by walrus01 »

canhascodez wrote: post #57
walrus01 wrote: post #32

Earlier quoted context omitted.

Counterpoint: Unless you've been in network engineering for an ISP with tens of thousands or more residential customers, you have no idea how much spam/abuse/trojaned PC traffic comes from residential users. There's a very good reason why ISPs block some of this, for the protection of their own users. It's even necessary to block port 139 windows file sharing or people will connect PCs with totally open file shares s…

OT: most developers have probably not been in network engineering for an ISP. If you've written anything about your experiences, or if you could be persuaded to, I believe that a lot of people on HN would be keenly interested (myself included).

I'd be totally willing to answer any specific questions. Not sure if I have the free time or motivation to write a full guide to network engineering, the best I can really do is point people at the right reference sources. I'm honestly not nearly as qualified to do so anyways, compared to far more senior people who work for huge transit ASNs.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#59

Post by technion »

walrus01 wrote: post #32
loup-vaillant wrote: post #26

Earlier quoted context omitted.

Dynamic IP is a bummer, but outright closing ports is a direct violation of net neutrality. It's baffling that it's tolerated at all. An ISP's job is to forward IP packets. Not read them. TCP/UDP ports number are the content of those IP packets—not the meta data. Filtering based on such content is already a form of discrimination. What's next, deep packet inspection? Some ISP filter by default, but they do this at th…

Counterpoint: Unless you've been in network engineering for an ISP with tens of thousands or more residential customers, you have no idea how much spam/abuse/trojaned PC traffic comes from residential users. There's a very good reason why ISPs block some of this, for the protection of their own users. It's even necessary to block port 139 windows file sharing or people will connect PCs with totally open file shares s…

I've done it at the scale of a few hundred and I still feel I've seen more than enough to 100% agree with you.

I've seen customers max out gigabit links with outbound spam from a compromised desktop. Contacting the customer leads to ignored emails and unanswered voicemails at best, angry Facebook rants about "getting in my face" at worst.

Re: How Spam Filtering Works: From SPF to DKIM to Blacklists

#60

Post by canhascodez »

walrus01 wrote: post #58
canhascodez wrote: post #57

Earlier quoted context omitted.

OT: most developers have probably not been in network engineering for an ISP. If you've written anything about your experiences, or if you could be persuaded to, I believe that a lot of people on HN would be keenly interested (myself included).

I'd be totally willing to answer any specific questions. Not sure if I have the free time or motivation to write a full guide to network engineering, the best I can really do is point people at the right reference sources. I'm honestly not nearly as qualified to do so anyways, compared to far more senior people who work for huge transit ASNs.

I probably mean more in the "human interest" relation of experiences sense than anything like a comprehensive guide to the profession. I'm not wholly ignorant of what happens to bits after they leave my router, but I know nothing about the people involved in making that happen, and I don't get the impression that it's widely written of. Anecdotes about daily challenges tend to be interesting; I would probably have gone quite a long time without ever knowing about port 139 otherwise I'm sure. But, I certainly wouldn't want to try to compel anyone to write anything they're weren't going to write anyway, so please don't take this as anything other than an expression of interest.
Post reply on HN