If two people wisphered secret messages between each other,should that be allowed? Should a policeman be privy to all whisperings? E2EE is just wisphering except much more efficient and can happen at large distances.
Should you be allowed to send mesages over snail mail using code words understood by only the recipient and no one else?
These are political questions. Governments having the authority to listen in on all private conversations implies they have that authority. Do you accept that authority where you are unable to express yourself to other humans without government employees logging and monitoring your expressions of thought? Maybe you really have nothing to hide now,but if ever you are given a reason to disagree and dissent with societal norms,your expressions of dissent will be monitored by the very people that have a lot to lose by allowing your thoughts to be expressed. If you can accept regulation of your speech and this authority over your life and liberty then it makes sense to oppose E2EE.
The problem is that the people whose communication is being monitored never accepted this authority,E2EE is just a way of enforcing my expectation that my communication to someone will be read only by that person. Removal of this right or privilege must be done via due process and full transparency without which justice and fairness would be very difficult.
Last point: E2EE prevents mass monitoring of communication. For warrantful intercepts,law enforcement benefits the most out of having access to the whole device. One approach would be to force a transparent backdoor that will side-load rootkits that come with a device specific certificate with a certificate transparency log maintained by a watchdog gov agency that enforces requirement of a warrant for each cert and criminal penalty for mis-issuing of certs or tampeting of CT logs. What if someone roots their phone and removes the backdoor? Make it illegal much like silencers and bullet-proof vests are illegal. It does sound very unpleadant and uncomfortable but much saner than weakening protocols. Like it or not you won't be able to convince elected politicians there is no way to securely gain access to a suspected criminal's phone even with a warrant.