I apologize for the strong language, my intention was not to quash discussion, but to stir it up, I have many thoughts on the topic but only a short time to articulate any of them- a spirited "I disagree!" was all I could budget time for at that moment. :)
I argue that part of the solution to this problem is the need for rules and standard practices. And moreso- the big problem and big deal here isn't even my own data, but rather the data other people gather on me. It's not that I want to keep my trove of My Little Pony erotic fanfiction secret- it's that when I do a Google search for incendiary dildos, I'm perfectly fine with google using my interest to target ads to me (they can inform me on the latest incendiary dildo technology!)- it's that I don't want my employer, the IRS, FBI, CIA, etc. to use that information to employ some hodge-podge machine learning algorithm on a dataset with my interest in incendiary dildos included (and therefore decide that they'll target me, because their algorithm indicate correlation between incendiary dildo interest and terrorism).
In other words, no amount of data siloing will protect you from this, dkarapetyan, it's your movements and actions in the digital world - and companies' observations of them - that we're talking about. There's simply no way (that I know of) for me to both communicate to Google my search terms and receive services, have them be able to use that data to target ads to me, but also have them not store that data on me in a way that a government entity could demand it from them. Of course, we're nervous about hackers from, say, drug cartels doing something similar, but we depend on responsible storage practices to stop that- we can presently safely stop the thief at the window, but the guy that shows up at the door demanding ransom or our family dies, we've got nothing. In that metaphor, because Google has to have access to the data in order to make it useful (and make money with it), someone with any kind of legal authority can send google's executives to jail until they comply and hand over the data.
The ransom metaphor may represent the crux of the whole debate. There will always be some risk of third parties gaining access to data, but some people use the 'implicit' argument to assert that because a bad actor could gain access to the data, that a government entity should be allowed to access/collect the data legally without a warrant. (That, by the way, is a more exact representation of the opinion that I called 'full of shit'.) The REAL risk, and the one where the ransom metaphor applies, is in an entity that can legally request such information and misuse it. Google, Microsoft, etc. can fragment and prevent most such truly valuable and extensive data collections from being breached fully but they are completely powerless against an aggressive intelligence agency insisting they pass on the data (and keep quiet about it) "or else".
Because the companies derive great value from our data, they want to keep it safe and use it for approved purposes- and license it or sell it to others who will do the same. We (and they) have tools to keep it reasonably safe for such purposes (not perfectly, but primarily with the idea that it would take many breaches to access all the data for even a portion of the customer database system). So we give them a meager amount of trust! But all those protections are NULL if the legal system allows government entities to legitimately demand all that data without probable cause.
Bah. Forgive the length. Brevity takes time, but especially where you'd asked me not to declare you full of shit, I wanted to respond. I appreciate your response, even though I disagree with you.