Bulletproof email infrastructure setup guide • Hacker News Board

Re: Bulletproof email infrastructure setup guide

#21

Post by krobertson » Thu, Mar 15, 2012, 6:01 PM UTC

igor_a wrote: post #12

mshafrir wrote: post #10

If I'm using a service such as Mailgun or Sendgrid, what parts of the article apply to me?

None, but all of them apply to Mailgun or Sendgrid:)

I wouldn't say none, you still need to setup your DNS settings with SPF/DKIM to support Mailgun/Sendgrid. They usually have documentation how to do that, but you need to be sure to do it. :)

Good article though... overall, if you aren't in the kind of email volume you guys are, just use Sendgrid/Mailgun. Sending email is a PITA and just pay someone else if you have the option. :)

Re: Bulletproof email infrastructure setup guide

#22

Post by cullenking » Thu, Mar 15, 2012, 6:38 PM UTC

igor_a wrote: post #3

cullenking wrote: post #2

Another option for bounce handling is to setup a client/server architecture. Continuing with the rails example: since directly spawning a rake task from a postfix hook will slaughter your machine if you get any decent amount of emails due to loading the whole rails stack each time, what I have chosen to do is this: Postfix pipes the bounced email through command line to a simple client that shoves the body over a soc…

I think this would be the next step for me. For now our performance isn't a big issue and bounces are processed in background cron job once per several minutes. For now volumes of bounces are not so high as you have, so we are just fine with spawning application. Thanks for commenting.

Cool - when we send out our newsletters to users, about 70,000 total active emails, we get a solid 1000 bounces in the first hour or so. Bounces decrease each time we newsletter, since we flag users with a bounce_count, and if it reaches 2, or the bounce is a 5xx (permanent error), we don't email them anymore. But, 1000 full stack loads in an hour on our mailserver would crush it!

Re: Bulletproof email infrastructure setup guide

#23

Post by igor_a » Thu, Mar 15, 2012, 7:25 PM UTC

cullenking wrote: post #22

igor_a wrote: post #3

Earlier quoted context omitted.

I think this would be the next step for me. For now our performance isn't a big issue and bounces are processed in background cron job once per several minutes. For now volumes of bounces are not so high as you have, so we are just fine with spawning application. Thanks for commenting.

Cool - when we send out our newsletters to users, about 70,000 total active emails, we get a solid 1000 bounces in the first hour or so. Bounces decrease each time we newsletter, since we flag users with a bounce_count, and if it reaches 2, or the bounce is a 5xx (permanent error), we don't email them anymore. But, 1000 full stack loads in an hour on our mailserver would crush it!

Nope, you've get me wrong - we start application every 5 minutes to process mailbox not for every email.

Re: Bulletproof email infrastructure setup guide

#24

Post by seanp2k2 » Thu, Mar 15, 2012, 8:57 PM UTC

icebraining wrote: post #6

How likely are servers to reject your emails simply because the PTR records don't match? I'm using my ISP's SMTP server as a smarthost and I've configured the SPF record to match their server, but making them add a PTR record is out of the question, unfortunately. (This is just for personal mail, that's why I'm avoiding paying for a proper setup)

Plenty. Protip from a sysadmin in the hosting industry: don't even try to run your own MTA unless you're just curious as to how hard reliable mail delivery is.

Use GMail or similar. Use iContact / StreamSend / MailChimp for newsletters. Use Google Apps for any business you start.

Re: Bulletproof email infrastructure setup guide

#25

Post by seanp2k2 » Thu, Mar 15, 2012, 9:00 PM UTC

cullenking wrote: post #2

Another option for bounce handling is to setup a client/server architecture. Continuing with the rails example: since directly spawning a rake task from a postfix hook will slaughter your machine if you get any decent amount of emails due to loading the whole rails stack each time, what I have chosen to do is this: Postfix pipes the bounced email through command line to a simple client that shoves the body over a soc…

>"Another option for bounce handling is to setup a client/server architecture." funny because e-mail already /is/ client-server architecture, but also server server architecture.

rube goldberg software design :\

Re: Bulletproof email infrastructure setup guide

#26

Post by seanp2k2 » Thu, Mar 15, 2012, 9:02 PM UTC

krobertson wrote: post #21

igor_a wrote: post #12

Earlier quoted context omitted.

None, but all of them apply to Mailgun or Sendgrid:)

I wouldn't say none, you still need to setup your DNS settings with SPF/DKIM to support Mailgun/Sendgrid. They usually have documentation how to do that, but you need to be sure to do it. :) Good article though... overall, if you aren't in the kind of email volume you guys are, just use Sendgrid/Mailgun. Sending email is a PITA and just pay someone else if you have the option. :)

PTR too. rDNS is important.

Re: Bulletproof email infrastructure setup guide

#27

Post by seanp2k2 » Thu, Mar 15, 2012, 9:04 PM UTC

igor_a wrote: post #20

gregbair wrote: post #17

I'd be careful about using the term "bulletproof" in this context. Before reading the article, I assumed it was talking about sending spam without getting blacklisted.

Read the article's disclaimer:) I think even if you'll have all possible things setup super-correctly and you'll put word `viagra` into the Subject field - nothing will help you.

mmmhmm. SpamAssassin will get those on almost every MTA out there.

Re: Bulletproof email infrastructure setup guide

#28

Post by pyre » Thu, Mar 15, 2012, 10:27 PM UTC

igor_a wrote: post #20

gregbair wrote: post #17

I'd be careful about using the term "bulletproof" in this context. Before reading the article, I assumed it was talking about sending spam without getting blacklisted.

Read the article's disclaimer:) I think even if you'll have all possible things setup super-correctly and you'll put word `viagra` into the Subject field - nothing will help you.

Would be interesting to get a perspective from whoever manages email for Pfizer corporate. How many spam filters out there must they have to battle against to get their legitimate email about Viagara through?

Re: Bulletproof email infrastructure setup guide

#29

Post by dredmorbius » Thu, Mar 15, 2012, 10:38 PM UTC

Great guide a couple of comments:

- DKIM, aside from buying you a "non-spam" pass (or at least more generous first treatment), can get you into "most favored sender" status with big email providers. Notably Yahoo, who, despite corporate troubles is still a huge player in hosted email.

- Even if you're using a mail service provider, setting up an SPF record for your domain, with your MSP's IPs (or better, an "include:" record to their sending SPF record) will help your mails go through.

- If you're using postfix, then you'll want to use 'qshape' and 'pflogsumm' to track your queues and delivery stats.

- VERPs are great. Until you run into an idiot recipient who insists on explicitly whitelisting all sender addresses. Including envelope sender. We at Krell Power have encountered ... a large energy sector company relying on Microsoft products who apparently favor this route. We've de-VERP'd their mails.

- Throttled delivery services with domain-based assignments are a great way to manage queues. Strong endorsement of that here.

- Track your mail reputation. Look up your domain and sending IPs on Senderbase, SpamHaus, Spamcop, and other reputation systems. Address complaints quickly.

Re: Bulletproof email infrastructure setup guide

#30

Post by icebraining » Thu, Mar 15, 2012, 11:11 PM UTC

seanp2k2 wrote: post #24

icebraining wrote: post #6

How likely are servers to reject your emails simply because the PTR records don't match? I'm using my ISP's SMTP server as a smarthost and I've configured the SPF record to match their server, but making them add a PTR record is out of the question, unfortunately. (This is just for personal mail, that's why I'm avoiding paying for a proper setup)

Plenty. Protip from a sysadmin in the hosting industry: don't even try to run your own MTA unless you're just curious as to how hard reliable mail delivery is. Use GMail or similar. Use iContact / StreamSend / MailChimp for newsletters. Use Google Apps for any business you start.

Well, the (sending) MTA is not exactly mine: I configured exim to send everything through my ISP's. Of course, the domain is still managed by me, and so is reception, although I keep Google Apps (free) for the backup MX.