My main objection to this is just the added complexity. Instead of a single suid binary that reads a config file and calls exec(), now you have one binary that runs as root and listens on a UNIX socket, and another that talks to a UNIX socket; both of them have to do asymmetric crypto stuff. It seems like the main argument against sudo/doas being presented is that you have a suid binary accessible to any user, and if…
Maybe also make /usr/bin/sudo immutable? would that help prevent a package manager from messing with it? I think so.
On Debian, for example, I have unattended-upgrades set up to automatically install security updates. sudo is reasonably likely to have updates for security reasons.