I have disabled PAM by default on all my boxes that run sshd for the last 9 years out of habit, I long ago forgot the reason why (probably because the gentoo sshd handbook entry said it was a good idea). Why UsePAM is set to yes in sshd_config by default on many distros is beyond me.
Key problem: aside from PAM, there is no way to use TOTP-based 2FA on your SSH server. And any modern security setup requires 2FA.
I think the problem is that somehow everyone ended up with PAM and are now somewhat stuck with it. FreeBSD too.