As a person who does run a number of (smallish) mail servers, some outgoing mail does get classified as SPAM and I have to fix it. It's very rare - every 3 months or so maybe, but it does happen. (Possibly the most irritating thing about this is they won't tell you what emails made them think you are sending spam, lets alone why they thing it's spam which makes root cause analysis damned near impossible.) The reverse happens too - we occasionally block non-spam emails.
But, and it's a damned big but, AFAICT, my email servers make less of those mistakes than the big email providers like Google or Microsoft, so the overall reliability of my home grow system is higher. It's got to the point that when someone complains they didn't get an email I got, my first question is "do you use gmail?".
The other big but it is it complex. It isn't just a single server - they are a network of them that distribute email across a geographically dispersed organisation. Despite being distributed it archives copies of all email that passes through it (as required by law here). There are some 2.5K of configuration lines driving it. So yes, it requires some effort to set it up. However, like your case those 2.5K lines haven't changed overly in over a decade.
The flexibility the system provides us in the way we handle email is invaluable. As a consequence of that flexibility a lot of emails are never seen by humans - they routed to a place they can be automatically processed.
Personally I wouldn't say email particularly hard, or particularly easy. It seems no more difficult than all the other things you have to do to manage a cluster of servers. The rewards for getting do it inhouse are pretty big, because just like everything else a programmer or sysadmin has control of, over time it will be scripted so heavily it will become almost invisible.