I am not saying that any of this is practical for 99.99% of email users. In fact, I would venture to say the opposite - my suggestions are practical for less than 10 in 100,000 users. With that caveat, there are a variety of ways to make email more secure.
Every Unix system has a mail daemon as a core function. If you've ever seen "This incident will be reported." because your user is not in the sudoers group, the incident gets reported via the local mail system. Mail servers today (outside of Microsoft's Exchange servers) are still overwhelmingly Unix-like systems. A mail user just needs a Unix host, a username and ssh + mutt/pine to access their mail in this way. Then an attacker needs to identify the unix host, the username and the ssh credentials. The first two bits are likely to be trivial - the Unix host is probably published in DNS MX records, and the username is probably the local part of the email address. The last part does not necessarily need to be a password. Good security practices will have it be an ssh keypair. ssh keypairs are provably computationally secure (until quantum computing comes along) and ssh private keys often also have passphrases. Other measures - like U2F security keys - can also be required as part of ssh authentication.
So this is probably the gold standard when it comes to security for an email inbox. Is there any way we can make this more usable so that a GUI mail client can be used? One answer is to run the mail client on the Unix host and use ssh X forwarding to display it on the client machine. Another answer is to run a POP or IMAP server on the Unix host, force TLS and use normal plaintext passwords over the TLS tunnel.
Both the ssh and the IMAPS solutions require some way to authenticate the server side of the connection. The ssh mechanism typically defaults to trust on first use, but since one is publishing DNS records anyways, a SSHFP (ssh fingerprint) record seems like a good additional step. The IMAPS solution requires re-authentication of the certificate for each session. I am not sure what the current state of the art is for this for IMAPS servers. I personally believe that the CA model used for HTTPS is severely flawed.
All of this to say that for the truly dedicated user of a certain minimum (but high!) level of technical sophistication, passwordless login everywhere is possible (ssh keys; or having the email client securely store the password for use with IMAPS).