Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Hostile Email Landscape

liminality.xyz

Post reply on HN
Print view View original

141–150 of 251 posts

Re: The Hostile Email Landscape

#141

Post by phit_ »

staunch wrote: post #56

This largely true and hugely disappointing. Our startup ( https://portal.cloud ) is making it possible for non-hackers to self-host their own email servers. It works very well for the most part, but we have had to explain to a number of them why their email sometimes gets bounced by the big proprietary cloud services. All of our users have their own domain names, IP addresses, SPF records, and correctly configured (a…

fyi: there's a little mixup on your main page, the $5/mo plan says 512gb ram instead of mb

Re: The Hostile Email Landscape

#142

Post by atmosx »

I did not realize that I am running my private SMTPd (+ imaps) for nearly 8 years. I never had issues. My score on test-mail is 9/10 because of lack of DKIM. I will implement DKIM, see if I can get 10/10.

Re: The Hostile Email Landscape

#143

Post by MortenK »

OP, I don't know if you are reading the comments here but in case you do: Don't get discouraged so quickly.

The reason this is happening is as the blurb from the MS postmaster help page: Your IP doesn't have a reputation yet.

The reason these rules are in place aren't about email monopoly, it's about spam. If anybody could setup a SMTP server and start firing off large amounts of mail, spam would be even more endemic than today.

You can configure your server perfectly, but that doesn't mean much, since it's your IP that's the problem.

If you have legit objectives, it's a pain in the ass for sure. But you are not the only one having this problem, and there's a solution for it.

All the big email service providers (ESP's) like Neolane, Exact Target, Mailchimp, Campaign monitor etc share this problem when they onboard a new client, who requires their own IP.

Deliverability is a surprisingly deep, technical topic, and all major ESP's have entire teams of specialists working on this.

If you want to make such a service as Fastmail, you need to get really into deliverability. It's not a walk in the park, but it's not impossible either.

I'm not a specialist in this particular area myself, so I can't give you that much specific advice. I've just worked elbow to elbow with a lot of these guys, so I know what kind of challenges they work with.

One thing I know for sure is really important, is the "warming up" of IP's. Basically the IP you are sending from needs to accumulate some reputation over a period of time, typically a month or two.

If you send out reasonably small amounts of mail to email addresses that exists and the recipients does not explicitly report you for junk mail, your IP get whitelisted and you will get a much higher delivery rate.

There's no quick fix unfortunately, and email reputation is hard to gain and fast to lose.

But it certainly can be done. You sound very competent on the server side of things, so to get your fastmail-like service up, I think it's just a matter of a bit more persistence and studying deliverability as a technical subject.

Hope this helps.

Re: The Hostile Email Landscape

#144

Post by lsc »

staunch wrote: post #138
lsc wrote: post #131

Earlier quoted context omitted.

Really? More than 99% of the junk email in my box is from "legitimate" senders who use their own mailservers. The truth of the matter is that it's already pretty easy to block the hit and run spammers. Most of the junk that gets through is sent by people with the resources to bully the spamfilters into accepting it.

Spam is viagra ads and the like (unsolicited commercial email). Promotional emails from companies you have a relationship with are a different thing. Users should be able to easily filter and ignore promotional email, but email hosting companies should never block it as spam. SMTP is not where email organization should happen. That's what email clients are for. Blocking DoS attacks and spam makes sense. Blocking Amaz…

>Users should be able to easily filter and ignore promotional email, but email hosting companies should never block it as spam.

See, you still seem to think that it's a negotiation between the senders and the receivers.

My point is that you don't have the right to the end-user's attention. If the end-user wants to hire someone to filter out messages they don't want to read, using whatever criteria for "don't want to read" that they like, this is the user's right.

(more relevant to the discussion, the email that is technically spam is pretty easy to block already, so a solution that requires a lot of social/political change is inferior to the technical measures we have in place.)

Re: The Hostile Email Landscape

#145

Post by teddyh »

rtehfm wrote: post #100
teddyh wrote: post #84

I sometimes see similar tales of woe, and I can only say that this does not match my experience. I’ve done this many times, you set up the mail server, configure DNS correctly (including reverse lookup), and that’s it. Never had problems being blacklisted or mail getting classified as spam. I suspect that people having trouble are sending a lot of mail , like “newletters”, etc. But I can’t prove this hypothesis.

Mail reputations are very real and pose an issue with mail servers. I had a gaming server for years and had many issues with Gmail, Microsoft and Yahoo, to name a few, filtering or blocking emails. Just last week, I setup a mail server using mail in a box on a new server I spun up on Digital Ocean using an IP that was on no blacklists and still had issues with sending emails to various Gmail subscribers. Even when I…

One more point of data from my experience: I have never set up mail servers at hosting providers in remote datacenters, only local servers in my own server rooms which I could physically touch. I guess it’s quite possible that Gmail, etc. have figured out by now where all the IP blocks of hosting providers are, and are very suspicious of them.

Re: The Hostile Email Landscape

#146

Post by unclebucknasty »

david-given wrote: post #111
unclebucknasty wrote: post #103

Earlier quoted context omitted.

No. Our recipients had no knowledge of the third party. Instead, their email providers contracted with the third party for filtering services. EDIT: Not sure why it would preclude their liability in any case though. Their customers would be paying to have spam blocked, not emails from legitimate companies with whom they've agreed to do business. If they were blocking these demonstrably legit businesses in error and r…

Right... their email providers, which they are paying to provide email from them . The money chain is still at your recipient's end. Note that I'm not suggesting you did the wrong thing here --- I don't see anything else you could have done. I'm just saying that pinning the entirety of the blame on the third party provider is wrong. ... Having read the rest of the thread: I'm sorry to say but the reason why you're ha…

>* pinning the entirety of the blame on the third party provider is wrong.*

We disagree. It's certainly understandable that the third-party might raise false-positives. However, what is wrong is when a business can demonstrate that it is not engaged in such practices, yet the third-party provider still refuses to cease penalizing them.

So, who else's fault would it be? Mine? The customer? The customer's e-mail provider who contracts with the third-party? Some unrelated spammer who makes the third-party's business really difficult to execute well? I don't think so. The third-party is in the business of profiting from the proper classification of senders. It's their risk and responsibility to fix these problems.

>the reason why you're getting hostile replies here.

Well, I didn't think the replies were particularly hostile. A little misguided maybe, but not hostile.

Anyway, I think you're also jumping to a lot of conclusions and, frankly, not reading very carefully.

Firstly, we're not an "email advertiser" in the sense you're implying. We're a member-based site with a newsletter, into which only members double opt-in out of their interest in our service. The ads I referenced are occasional sponsorships by businesses that are also highly-relevant to our service and members' interests. But, we don't just push ads and we follow all rules regarding proper email management.

Secondly, our content was never the issue anyway. As I mentioned, the issue was that we were falsely claimed to have been sending to honeypot addresses and/or defunct addresses. I enumerated the reasons why we knew this to be in error.

I hate spam as much as anyone. In fact, my reaction to it is probably borderline irrational. But, there are legitimate businesses with legitimate customers between whom e-mail is a mutually beneficial channel. There's nothing "scary" about that. And, my initial point was in response to an ancestor comment, wherein I agreed that certain "anti-spam" practices make it far too difficult for those who use the channel properly and respectfully.

Re: The Hostile Email Landscape

#147

Post by zrm »

I was thinking about this a while ago and have been meaning to write it up and post it somewhere, so I guess this is as good a time as any.

Hashcash (also known as the precursor to Bitcoin) was proposed to solve this problem in 1997:

https://en.wikipedia.org/wiki/Hashcash

The trouble with it is that it requires computation for each sent message, which is bad for senders with low resource devices or legitimate mailing lists. I want to propose a variant.

Instead of creating an expensive hash against the message, create an even more expensive hash against the (sender TLS certificate, receiver domain name) pair. This implies using TLS but it works just as well even if the certificate is self-signed. Then each mail server only has to generate a hash once per recipient domain, ever. Every message that mail server sends to that domain is tagged with that hash. A legitimate mail server will have already computed hashes for all the domains its users regularly correspond with and rarely if ever need to do any more expensive computations.

If spammers do the same thing then the receiving server can mark all messages sent with that hash as spam. So there is a highly disproportionate cost to spammers (even if they have more computing power) because to avoid that they have to continuously generate expensive new hashes. Which can be made arbitrarily expensive because legitimate servers only need to do it once. And a new hash is much less valuable to a spammer than a domain name or IP address is today because each hash can only be used against one recipient domain. The required amount of computation can be set by the receiving server so domains with more users can require more computation.

If a legitimate mail server is compromised by a spammer then it will have to generate all new hashes (because the spammer will presumably immediately ruin the reputation of the compromised ones), but the reputation of the legitimate sender's email domains is unharmed because the reputation is tied to the hash computation, not the sender's domain name(s).

And adding support to mail servers would require no configuration whatsoever. You install server version N+1 and it starts tagging outgoing messages with hashes that receiving servers can verify.

So here's the traditional form: http://craphound.com/spamsolutions.txt

How'd I do?

Re: The Hostile Email Landscape

#148

Post by jcrites »

tacon wrote: post #97

I've managed my own mail server since 1993, and my email address has been the same that entire time. Here are some tips for maintaining sanity: Greylisting still works amazingly well. With a long, long whitelist and greylisting plus DNSBL, I don't even bother running a spam filter, since the little bit of spam and emails from new senders ends up in its own directory as it came from a non-whitelisted sender. Comcast f…

> suddenly Gmail was connecting to my mail server over an IPv6 interface

Nitpick: when sending email, the sending mail server establishes a connection to the receiving server, not vice versa. The sender is responsible for looking up the domain name's MX record, choosing a hostname from those listed, initiating a connection, establishing an SMTP conversation, and negotiating protocol enhancements like SSL.

That said, I could see a change in behavior like this resulting from action on the receiver's side. Though the sending server makes the choice of how to connect, that choice can be influenced by the presentation of the MX record. I haven't specifically run into this before, but I could see this happening if, for example, the MX record advertised a host with an A record first (IPv4), and then was changed to list a host with only an AAAA record first (IPv6).

There's quite a lot of arcana in the email space that's difficult to manage without careful, deliberate effort. Sending email is harder than it should be, and a lot of that difficulty results from anti-spam or general anti-abuse efforts. If it was easy to bring a server up and begin sending with a new domain name and IP address while successfully reaching the inbox, then spammers would do exactly that. So new identities are put through the wringer to prove they're legitimate. Well-known platforms have an easier time sending email because they can relay traffic from infrastructure that the ISPs already trust; and in turn they maintain that trust by keeping abuse off their platform.

Re: The Hostile Email Landscape

#149

Post by rokhayakebe »

waynecochran wrote: post #82
rokhayakebe wrote: post #64

Earlier quoted context omitted.

The minute you say bitcoin is the moment you cut out 99% of the population; unless it is in the background and one does not have to interact with it directly. Other than that, I agree paying to send will reduce spam. But also look at your postal mail box. There is arguably more spam there than in your digital inbox, and that one costs (stamps).

Yes, but if the email is digitally signed I can be certain who sent it (at least the address) and, unlike postal mail, I can electronically sort it. Email from folks I know would receive the highest priority. Are folks afraid of bitcoin? What can be worse than just having a 16 digit number printed on a card -- that is a 1000x more vulnerable.

Are folks afraid of bitcoin?

You are part of the tiny percentage of the tiny percentage of the population that understands bitcoin.

Re: The Hostile Email Landscape

#150

Post by unclebucknasty »

lsc wrote: post #125
unclebucknasty wrote: post #102

Earlier quoted context omitted.

No. Our customers have email accounts with providers like Comcast. Comcast uses a third-party filtering service to handle spam. So, our customers had no knowledge of the third-party, and some contacted our support staff to question why they weren't receiving our emails. And, of course we attempted to identify the problem with the third party. If you read more carefully, you'll find that in my comment. Only after they…

>And, of course we attempted to identify the problem with the third party. If you read more carefully, you'll find that in my comment. Only after they insisted their classification was correct and refused to remove us from the list did we explore our legal options. My point here is that the recipients of the email and the senders of the email have very different ideas as to what constitutes a "correct evaluation." -…

You are really missing the point here and you have some key assumptions wrong.

>recipients of the email and the senders of the email have very different ideas

What does that mean? They sign up for our service and expect to get e-mails from us, which they don't receive. We have the same "idea", but the third-party is interfering with that "idea".

>if the recipients can't complain, that's a problem

I will repeat that by definition of the business model, the recipients don't complain to the third-party because they have no knowledge that the third-party even exists.

>but it's not like people don't have a choice of email providers

Even if recipients did know about the third party, do you know the switching costs of e-mail? And, you're suggesting that we should wait until all of our affected members get fed up and change providers (hoping that provider doesn't use the same service, etc.)? All of this, when the problem is obviously being caused by the third-party's error, yet they have no responsibility?

Sorry, but of course the third-party is absolutely compelled to provide some recourse for legitimate senders that are mis-classified. Too hard with all the spammers out there, you say? Sorry. Pick another business.

Post reply on HN