I have a mailserver handling some personal email, but I feel it's too risky (to take the responsibility) and too much effort to host email accounts for just a few other people. You can and probably will be every once in a while blacklisted by one of the big providers or have legitimate email bounce, even if you have SPF, DKIM, TLS and your own spam filters set up. You also have to keep an eye on your servers to see if no new filth gets through. And you'd probably want to keep a backup relay ready. You have to provide ways for the users to configure or fine-tune their individual spam settings and mark messages. You most likely want to install a web interface next to the IMAP and/or POP service, which opens another can of worms.
I feel I'm too old now – meaning I have so many other responsibilities – that I don't want to babysit something that is after all rather crucial and should "just work". If you have the energy and time, please go for it, otherwise just search for a reliable paid service.
Note that assuming privacy when talking about email, even though most protocol interactions might be encrypted these days, is in my opinion somewhat misguided. Don't use email if it's truly private. Or use end-to-end encryption, such as PGP.
There might be a hole in the market for a company that helps geeks host reliable email servers, for those that want more control than just an IMAP account with sieve support, but maybe the margins are too low and fighting spamming subscribers too hard.