Government will always be a major threat vector to the population. Encypted communication tools have the effect of creating a wider distribution of power which lessens that threat. I think the trade-off of denying the establishment some control that would empower it to more effectively combat malevolent behavior by fringe elements, in order to better protect the majority from potential abuses and structural exploitat…
That's all this is about: cost. It's more expensive and the system would have to dedicate more resources to fewer targets, being forced to prioritize their efforts.
Sure, that means that investigating truly bad people will be more difficult. But, that extra cost means that the average well-meaning person won't be nearly as vulnerable to bad actors who use Total Information Awareness for nefarious purposes.
This shouldn't really be controversial now, in-fact it's a well established principle in criminal law called Blackstone's Ratio.
"It is better that ten guilty persons escape than that one innocent suffer."