— the key retention is the biggest issue. You need to keep your key around for a long time, probably storing copies of it. This increases the probability of a leak.
— there is no method to revoke a key with a 100% assurance that nobody will use or trust it afterwards.
— if a key is broken or leaked, every encrypted message you ever received can be deciphered. And I hope you realise it, or future messages are also at risk.
Those are the three major flaws I can remember right now. The way people usually use it make them feel safe while they are not necessarily. It's a bit like reusing a complex password on different websites (although far less critical since the key is assymetric).
The way I use it now (and I never actually needed it, to be fair), is that if some third party want to send me some confidential information, they request my public key via an open channel, and I then generate a key, unique for this conversation (ideally, it would be done on a per-email basis). Of course, the complex part here is ensuring they receive the proper key (ie, no man in the middle). This can be done by using a side (preferably secure) channel.
Of course, in today's world, encrypted emails are not the best way to communicate privately, in my opinion, but that's another story.
And there are a couple of other issues regarding encrypted emails such as its adoption, it's complexity, etc. But none as fundamental.