I do this. It's not easy, but it's not as bad as most of the comments here would lead you to believe. If you have the wherewithal to run things on a server it's definitely within reach, albeit probably not a cost-effective way to spend your time.
Let me make a few recommendations. I'm not a sysadmin by trade but I know a few things you can do that will improve your lot in life running mail.
- Postfix and Dovecot. They have the right combination of ease of use, power and security. Don't overconfigure; try to get it working with basic settings first and then evolve it towards what you dream of rather than setting out to configure it that way first.
- SPF and DKIM. I have no trouble getting messages to Gmail and I think this is part of why.
- Make sure your hosting provider is not huge and very high quality. I chose RootBSD because they're small but highly technical. If you have a lot of spare cash, iNetU are quite good and sometimes help with FreeBSD. The larger or crappier the host is, the more likely you'll wind up in blacklisted IP space. (BSD hosting companies tend to be smaller and more technical, and BSD is great software, so I'd recommend that if you're interested.) Getting off a blacklist isn't a lot of fun and it's not hard to wind up on one, but I find being on a discriminating host is a good preventative measure.
- Rely on IMAP. If you want webmail, try and find one that is really just an IMAP frontend. I tried and liked Roundcube a while back; these days I have IMAP clients everywhere so I don't know what the new hot stuff is, but IMAP is fantastic.
- I strongly recommend you get an account with DNSReport.com. Their software can detect most of the DNS problems you can get yourself into that wreak havoc with mail. Odds are good you'll be doing a lot more DNS than before, it's a great tool to have in the toolbox.
- Stay on top of your security updates. I recommend running sshguard and whatever other security software/IDS/firewall type stuff you can stand. Make sure you're not giving out a bunch of shell accounts with root on this server. Seems obvious, but people forget or get lazy. FreeBSD will email you a security message every day; if something like that isn't coming your way, consider trying to set it up. It tells me, among other things, who tried to log into the server, how many times they failed, what their IP was, and lots of other stuff.
There are a number of nice upsides to running your own mail server.
- Email can be hooked up to the database various ways.
- Automatic emailing for free (keep an eye on it).
- Scripted email handling for free (Procmail etc).
- Get system-generated messages emailed (Nagios/monit etc., login/sudo failures, etc.)
- Advanced forwarding/wildcard accounts.
Anyway, I hope you do give it a shot despite the nay-saying. Cost-effective? No, but it's a blast, and many of the upsides would be hard to replicate with Gmail. Of course the web mail UI will be worse. Tradeoffs.