Earlier quoted context omitted.
It's a cost-benefit tradeoff like anything else. Unfortunately, the costs are filtered through fear-mongering and the benefits are trade secrets, so the debate is particularly ineffectual. The thing I find most aggravating about it is that the standard for harm for data seems to be "What could a totalitarian government do with it?" and there are very few useful things that couldn't be used for very bad things in the…
Part of the problem is that privacy, at least in this context, is framed as a "debate" at all, rather than as a matter for individual choice. Some people would gladly share medical information for the greater good if asked. You'd have to pry it from the cold, dead hands of others. The problem is that we currently don't separate those who would like to provide their information from those who would like to refuse beca…
There isn't any coercion going on with any of the things we're talking about. Every technology product has at least the choice to not use it.
It's quite difficult to ask meaningful questions about what users are comfortable with, get meaningful answers, and then figure out how the answer they've already given applies to a grey area situation where the cost of getting it wrong is a lawsuit. It becomes no longer sufficient to treat the data with respect and only use it for beneficial and privacy respecting purposes. You now have to constrain it by another set of rules whose relationship to what's actually happening can be unclear and arbitrary.
Some products work without storing any data. Most don't. For those that require user data to fulfill their basic function, the engineering cost of exempting certain data from certain systems can be much higher. One programmer screwing up becomes a lawsuit.
This is essentially the situation with HIPAA. Everyone is too worried about liability to do anything innovative, so that sector doesn't improve.
If someone is actually harmed by something a company does with user data, then it's entirely appropriate to stop patronizing that company, or to claim damages through all the normal routes. The presumption of not trusting anyone with data in advance of any actual harm is what I object to. Data can do real and permanent good in the world, and some companies are worth trusting (particularly since all of their incentives are to remain trustworthy if they want to continue to exist.)