This is a late reply, but I'd been thinking on what you said and wanted to figure out where my disagreement lay. I think the biggest issue is that scale must be considered, and not merely whether each individual action is justified.
* Asking if somebody saw Brian at the bar last night is acceptable. Asking everybody if they saw Brian somewhere is acceptable in limited circumstances, such as Brian having been kidnapped. Asking everybody to list out who they saw and when is an unacceptable violation of privacy.
* Street photography is acceptable. Taking a picture once every day of the same house may be acceptable, such as if it is a historically interesting building, or if it is your own house. Collecting millions of street photographs, along with the time they were taken and who was in each photograph, is an unacceptable violation of privacy.
* CCTV is acceptable. Maintaining records of CCTV indefinitely may be acceptable, depending on the type of building. (e.g. The Pentagon may be justified in keeping CCTV recordings indefinitely, but the local laundromat is not.) Linking CCTV cameras together into a centralized query-able network that tracks people between locations is an unacceptable violation of privacy.
There are two common features that I realized in these examples. First, even if an individual action is acceptable and justified, repetition and coordination of that action may not be. This is similar to how saying hello when crossing paths with somebody is a courtesy and a pleasantry, but saying hello to the same person every 30 seconds is stalking. The difference in scale produces a different in outcome.
Second, there exist gradations of privacy, rather than being a binary divide between public and private spaces. Between a completely private space, such as a person inside their own home with the windows drawn, and a completely public space, such as a person giving a speech on live TV, there are intermediate spaces. A person who is walking down the street has partial privacy, where their actions may be remembered by passersby for a day or two, but wouldn't be remembered a month or a year later. In the past, these gradations of privacy were maintained by the limits of human memory and the high cost of technological memory, but the cost of technological memory has fallen to a point where this social construct is breaking down.
Summing up, I would say that we cannot have a modern technological society if we *don't* treat tracking with the same concern we treat medical consent.