I run my own mailserver and I've been doing that for the last ~10 years.
I have a static ipv4 at home, and my biggest problems are the following:
- residential providers won't delegate a reverse dns zone or set a reverse ptr record for you -- or at least my ISP (Fastweb) will not do it for a residential contract
- spam lists (spamhaus etc) will blacklist residential ipv4 pools by default applying what effectively is a prejudice (and defamation).
Some considerations:
Major providers will effectively do as much as they can to prevent other organizations (let alone individuals) deliver their own mail.
Google is particularly shitty in this regard: it regularly delivers my mail to spam despite having both SPF, DKIM and DMARC in order. My gmail inbox however is full of SPAM because google decided that I really have to look at those promotional email.
Microsoft is surprisingly good instead: upon rejecting mail initally they're going to direct you to an automated procedure to de-list your domain from their spam services, and it works. Kudos to microsoft.
SPAM, surprisingly, isn't really a problem: if you have sufficient checks for incoming mail (does SPF for the domain allows this ip to send email? do DKIM signatures check out? are they using SSL/TLS for their connection? et similiar) you basically won't receive spam.
Most OS vendors (Red Hat / Debian / Ubuntu) deliver postfix with poor cryptographical default settings, meaning that the default settings will connect to plaintext SMTP to deliver outgoing email and will not setup submission (tls) by default, not even with a self-signed certificate. In the time of letsencrypt being available, this is a dumb choice.
Running low-volume mailserver is surprisingly light on resources. A raspberry pi is likely overpowered for the task. You can use pretty much anything and it's going to work, as long as it powers on and doesn't lose data.
Running a mailserver is also surprisingly versatile. The possibilities are pretty much endless. You want to alter outgoing emails? No problem, look at PCRE maps and postfix's header_checks (or similar). Want to have mail aliases? no problem. Want a catch-all address? easy. Want to hook a service into mail delivery chain? look at the milter protocols.