I have not done so in the traditional sense in quite some years. My experience was that it was an increasing headache due to crashing into a wide variety of anti-spam efforts. Get email past one and crash into another.

Depending upon your use case – using the “forward to a smarthost” feature in some mail server packages to forward to a mailserver run by a SMTP service provider with whom you have an account might work for you. Then it still looks to local software like you have a local mailserver.

If I were going to do a conventional, no-smarthost mailserver today, I think that I would probably start out by setting up a bunch of spam-filtering stuff — SpamAssassin, I dunno what-all gets used these days on a “regular” account — and then emailing stuff from my server and seeing what throws up red flags. That’d let me actually see the scoring and stuff that’s killing email. Once I had it as clean as I could get it, I’d get a variety of people I know on different mail servers and ask them to respond back to a test email, and see what made it out.