Live data from Hacker News

After self-hosting my email for twenty-three years I have thrown in the towel

cfenollosa.com

731–740 of 744 posts

Re: After self-hosting my email for twenty-three years I have thrown in the towel

#731
post #730

Earlier quoted context omitted.

> Any guess on why people's experiences with spam are so diverse? That would be an interesting research project. The domains getting most spams on my servers are the ones that are old (20+ years) which I guess makes sense. Checking the rspamd logs for the last month gives just shy of 8500 emails with what I'd consider "definitely a spam" score. There's probably another 1000-1500 sneaking under that.

> > Any guess on why people's experiences with spam are so diverse? > That would be an interesting research project. Indeed. When you say get spam, do you mean pre or post-filtering? Over the last 30 days (I don't keep them longer than that) I've received 43 spam emails which were sent to the spam folder (so I wouldn't ever see these, other than because I went and looked now for the sake of this discussion). In the s…

> Indeed. When you say get spam, do you mean pre or post-filtering?

All my numbers are post-rspamd. There's still sieve after that which does the "is it scored more than X?" to redirect into spam/not-spam.

> There's a fair amount of would-be spam that gets blocked during the SMTP transaction

Yeah, the blocklists and protocol strictness rules definitely cut down a lot before it even gets to rspamd.

Re: After self-hosting my email for twenty-three years I have thrown in the towel

#732

Earlier quoted context omitted.

I think everyone knows this who owns a domain. Probably people say this because (like I do too) they have all the control over the name. If I rent an apartment, I do have to consult the owner to make major changes.

I think people who get their email on their own domain generally consider it as their permanent home and don’t think about what will happen when the domain no longer is registered to them. Using the apartment rental analogy works up to a point. With an apartment you can get the postal service to forward your mail to a new address. With a domain name there is no way to get this to happen. Whoever holds the registratio…

In case of death it's indeed an issue if someone isn't planning ahead.

With grace periods of the domain expiry process it's quite safe. People have plenty of time to recover it. It's also hard not to notice if your mail stops arriving.

Re: After self-hosting my email for twenty-three years I have thrown in the towel

#733

Earlier quoted context omitted.

By lot of admin you mean that it's not automated? That's sad indeed.

> By lot of admin you mean that it's not automated? That's sad indeed. How would I automate "[someone with an account] just entered abc-xyz@domain into a web form to subscribe to something, add that to the valid alias file"?

As I understand, spam filters like postscreen are designed as reverse proxies or relay servers. IIRC spamgourmet supports your scenario.

Re: After self-hosting my email for twenty-three years I have thrown in the towel

#734
post #643

Earlier quoted context omitted.

The overwhelming majority of my legitimate mail has seen consistent and irrecoverable 14 day blocks for the last decade+. One on an IP that only did my email for over 6 years. Yet the oligopoly started to block blocks of IPs, including my IP, without recourse. If you say this problem doesn't exist, you are either not familiar with the problem, misunderstanding it, or simply lying. I hope it is just miscommunication.…

Exceptions to the rule do happen; I'm sorry your experience was the exception. Dealing with that kind of thing can be quite a pain if you're not experienced with it. But again: "If you find yourself in an IP range involved in a severe, ongoing, high-volume spam scenario that's affecting your delivery, then it means your provider is not managing IP range reputation, or not doing it very well, and you should vote with…

Did that. Three times. It's whackamole.

You know why? Because any IP-adress that I can afford, is going to be blackholed.

You make it sound like the blame lies with the Linodes, Digital oceans or even the Hezners or ISPs. This is not their fault. The blame lies, entirely, with Microsoft and Google (And to lesser extend Yahoo) using a cannon to shoot a mosquito.

Again: My IP (the address, not the range) was fine. It had been fine for many years. Why then, must Google and/or Microsoft, randomly, block this address? Why can't they make exceptions for reputable addresses within a range of bad ones? (I know why: they are lazy and use the easy path: just block everything and accept some "collateral damage", especially when that "collateral damage" cements their oligopoli a bit more, and when avoiding that collateral damage not only costs more work, but enables competition to exist (and grow))

Re: After self-hosting my email for twenty-three years I have thrown in the towel

#735

Earlier quoted context omitted.

> By lot of admin you mean that it's not automated? That's sad indeed. How would I automate "[someone with an account] just entered abc-xyz@domain into a web form to subscribe to something, add that to the valid alias file"?

As I understand, spam filters like postscreen are designed as reverse proxies or relay servers. IIRC spamgourmet supports your scenario.

> IIRC spamgourmet supports your scenario.

Not really - that's a limited use alias. Doesn't really work for when you want to keep getting email from places. Also if I'm reading that correctly, spammers can just send mail to .20.@spamgourmet.com and you can't protect against those 20 spams.

Re: After self-hosting my email for twenty-three years I have thrown in the towel

#736
post #383
post #243

Earlier quoted context omitted.

Wow that's ingenious - and obvious when you think about it. I guess evolving the email standard is a much bigger obstacle though, no matter how clever the proposal.

it's ridiculously easy to create a template that is stored and when the remote client pulls the message only a few variables has to be substituted. it's how spam is generated after all :)

Yes, but in the context of Freenet specifically, there's a lot of encryption and hashing involved in every transaction, which are computationally expensive, and there's no CGI equivalent at all (the whole network is basically static, append-only content addressed with the content's hash). So to create many messages you'd need to first generate the content, then encrypt and hash it, then insert it into the network starting at your node, then provide signature and checksum verification on every request. After some time, if the content is widely requested, some other nodes will cache it, but you can't really expect spam to propagate via web of trust; it's more probable that it will be added to spam filters which, themselves, will propagate. As everything is signed, you'd need to create a fresh node for each spam, but that means your new identity is not trusted by anyone - you'd need to convince at least some users that you're a user that won't abuse the system. Only to get your whole identity blacklisted once you do abuse it.

It wasn't perfect, and Freenet's usage was a huge PITA from the usability/ux perspective, but in a constrained environment like that this change in the protocol had much bigger impact than it would on the open web. Not sure how it turned out, but the trend until I left the community was very positive with regards to spam.

Re: After self-hosting my email for twenty-three years I have thrown in the towel

#737
post #684

Earlier quoted context omitted.

So normally in the cases like this, you also have to tell Google about this, and you typically do this by using one of their paid-for products like workspaces or apps for business or whatever they call it. So let's say that you decide to host skywall@skywall.example with Google. You pay them for workspaces and you likely tell them "I would like to use this domain I already have, with email". They then tell you "OK, a…

I learnt about delv for the first time reading your comment and struggled [1] to get the commands you used here to work on my Mac. Turns out that I have to treat Apple-included binaries of OSS utilities with suspicion and instead use the version maintained by the Homebrew guys. 1: If anyone else is interested, I wrote a short article about it https://ayewo.com/how-to-get-delv-working-on-macos/

delv is the successor to dig, and both are tools that come as part of BIND the DNS server for making queries and debugging DNS. Delv happens to understand DNSSEC a bit better, so if you ever need to debug that, it is handy to have.

In this case dig or delv would be fine. You can also use https://mxtoolbox.com/SuperTool.aspx and pick "MX lookup" to find the MX records. It will also resolve the IPs, both IPv4 and IPv6 (A and AAAA) for you for the returned records.

Their supertool also breaks down SPF records, for example, into their meaning and tells you if they're valid or not. I think other links were posted that do similar things but I haven't had a chance to try them yet.

I've mentioned reverse pointers in various cases, delv can do those lookups too. Here is how you do it: first you find the IP of say mail-ed1-x529.google.com, which happens to be a mailserver of Google that was the last step before my mail server when sending myself a message from gmail. For a bit of a change, this is an IPv6 only server, and it has IP 2a00:1450:4864:20::529. So let's look it up with -x:

    delv -x 2a00:1450:4864:20::529
    ; unsigned answer
    9.2.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.ip6.arpa. 43200 IN PTR mail-ed1-x529.google.com.
There are some special domains of the format .in-addr.arpa and .ip6.arpa that point (PTR) to the DNS name. dig/delv know how to reverse and query appropriately. You can use this for precisely this reason: to look up the intended host. This makes for a nice sanity check during email sending, because in SMTP I can announce myself as any host I like, but the recipient can do a reverse lookup on my IP and see if I'm telling the truth.

One of, but my no means the only, reasons you can't send email from a home connection is because you announce yourself as smtp.zahllos.example but your reverse dns resolves to something like .dynamic.residential.isp.com. You can find out yours: type "what is my ip" into google, copy-paste that and do a delv -x on it and see what your "hostname" is according to the internet.

Re: After self-hosting my email for twenty-three years I have thrown in the towel

#738
I think the author of the article makes a few mistakes in his diagnostic.

You should never use a service where you cannot report a problem or where the administrators are out of reach. Of course it includes the "big techs" but they're not alone.

The underlying issue is that most of them delegate their filtering decisions to third parties. And many of them rely on the same centralized IP and domain blacklists.

Because of how the filtering is done, the End User generally never discovers that any filtering is happening. Only the sender may be notified by his mail relay of the delivery failure.

Of course, the blacklisters are not going to put the IP addresses of the big Email Vendors into their lists, if they did, millions of people would be notified of delivery failures with the risk of them discovering who is responsible.

""" Unfortunately, the computing power required to filter millions of emails per minute is huge. That's why the email industry has chosen a shortcut to reduce that cost. """

Even 20 years ago the computing power wasn't a problem. He probably has this impression because he's been using SpamAssassin. The real reason for why they are taking the "shortcut" is carelessness toward their service and users.

The excuse of saying that you should block messages before delivering them because it takes disk space is also heavily promoted by the blacklisters. Indeed, if the message was simply delivered to a Spam folder with the actual reason for which it has been classified as such, users could discover who is responsible for the filtering.

The good news is that there is some success in getting the big email services to remove centralized blacklists.

Another problem is that too many administrators of smaller services are not even aware of their reliance on blacklists. Sometimes this is because they have used an easy installation script for convenience, or because they've copy/pasted a configuration. And of course, there are those who do not understand the ethical implications of doing such a thing or are just foolish.

""" So, starting today, the MX records of my personal domain no longer point to the IP of my personal server. They now point to one of the Big Email Providers. """

This doesn't make any sense. The MX records are for inbound, not outbound, he could have used a different relay for sending mails yet still use his own relay for receiving (perhaps he doesn't know that?). Instead, he switches to a provider that is known for contributing to the problem. This is... disturbing.

Re: After self-hosting my email for twenty-three years I have thrown in the towel

#739
post #704

Earlier quoted context omitted.

Amazing — even though you quoted me, you still managed to reply to something I didn't say!

You say that self-hosting a mail server is obvious and extremely low effort, which simply is not true. But ok, let's say that you happen to have a clean IP and no spam or dmarc issues. You still had to choose what to deploy, which is not obvious and actually setup/maintain your server which is not extremely low effort.

Please read the comment you are replying to again. I said that it's obvious that it's worth doing, not that the work to set it up is "obvious".

Of course you need to learn some things, but it's not that much work. If you're the type that enjoys learning this sort of thing, you'll have a good time. If you're not, don't do it!

Re: After self-hosting my email for twenty-three years I have thrown in the towel

#740
post #710

Earlier quoted context omitted.

sure, there is no problem with having 95+ % or more handled by professional industrial companies the problem is that they literally don't even allow others at the table, only way is to force yourself a seat by having enough traffic

Is it getting harder to self-host? Yes. Impossible? No. Most senders just find it easier to pay for it.

Everything is getting harder as tech gets more and more complex. (As IT matures.) That's okay. The problem is that the incentives suck and there's too much business in email. The big ones have built their moats, and there's not much to do.

"SMTP" was simply never made to be resistant to this. (Nor the web in general.)

Post reply on HN