An email provider would need a "to" code from you when you registered with them. Then with each request to download your mail, they send you a randomly generated number encoded with your "to" code, and you respond by decoding it and sending the number back encrypted with another number they give you.
They don't have to know where the mail is actually going, or what is in it. Just that they are getting a request from a person who owns that mail.
It gets more complicated than that, for the wonderful reason that there are so many more great things you can once you have standard primitives for encryption and cryptography. For instance, instead of a general email address, you can also give unique addresses to people so that only people you want can send you anything, and you can verify who its from.
You can still have a public address just by creating and publicizing an address for that purpose.
Once their is a standard library that developers can take for granted there are so many other things that can be easily done.
Today many of those things are difficult simply because there is no standard library, and everyone rolling their own implementations creates a nightmare. It is the vetting and fixing every single vulnerability from each implementation that is too difficult for most developers, even very good ones. That is one reason why everything is so insecure today.
A first draft of how the encryption works for email would probably one day's white boarding, since the primitives for this kind of thing have been around a long time. And people have done this kind of thing many times before in small contexts.
The actual implementation, to be very fast, extremely vetted (even formally verified) for correctness and after iterating many times of possible use cases and corner cases, would take some time.
There wouldn't need to be a single code base, but the point is to have very few implementations that are extremely well vet. So few implementations creates better security.
That way most applications and users can forget its even there.