> My argument would be that they need to find a way to access info on specific individuals as opposed to grabbing data indiscriminately from the pipes. I don't know if this is possible but it's the job of the IC to come up with a solution compatible with the law and human rights.
Well this is close to what is happening. As I understand it, to indefinitely store data the system would need to be tasked to track certain targets of interest. That said, I would be curious to see the success stories for data derived from specific targeting versus those which required past data that was only available due to the "rolling buffer" of data/metadata (I am assuming that this is what you're referring to when you say "indiscriminately"). The "rolling buffer" setup could not have been very easy to create versus the "retain data only for certain targets and drop other data" methodology, so there must have been some intelligence need for it at some point, but with the prevalence of HTTPS I'm not so sure it'd be useful. Could be wrong though, surely at least an interesting way this could be reconciled.
> As for the USA being in the dark I think my points are relevant to most countries so the idea is not to disadvantage anyone but that there is a level playing field that respects everyone's rights. Obviously some countries are going to violate human rights but it shouldn't be highly developed democratic nations just because the right is less glamorous than others.
I just find it incredibly hard to believe that nearly all other countries are not performing similar forms of SIGINT collection in the same way we are. I believe that it is just USA and the UK who are in the spotlight for this due to the leaks in 2013. That said, I absolutely admit that I don't have much to stand on without the ability to point to public evidence of this.
> It's very hard to have a useful debate on this when we don't actually have data showing how useful bulk data collection is.
You are absolutely correct here, the current "trust us" reasoning is unhelpful. Solid statements such as "By performing bulk monitoring of internet traffic flows, we were able to task the system to capture all traffic from selectors (phone number, e-mail, IP, etc) associated with BadGuy X1 X2 and X3, who were planning Y attack on Z and we were able to have AlliedMilitary thwart it" would be a far more productive way to have a clear discussion on the topic at hand. The IC has an aversion to publicizing information regarding this sort of thing presumably due to fear that enemies would figure out how they got the required intelligence and then change their tradecraft accordingly. I think that concern is a valid one as well, and I truly hope that we can figure out a decent way to reconcile it so there can be a middle ground. Perhaps it could even be helpful to provide aggregate information at an unclassified level and have cleared representatives publicly confirm the accuracy by reviewing the case by case (specific) classified details.