Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

arxiv.org

Post reply on HN
Print view View original

11–20 of 42 posts

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#11

Post by pfortuny »

quantisan wrote: post #4
mirimir wrote: post #2

Could someone please ELI5 how an "intimate" provider (such as Apple, Google or Microsoft) can collect any data ongoingly without eventual loss of privacy?

Let’s say you wanted to count how many of your online friends were dogs, while respecting the maxim that, on the Internet, nobody should know you’re a dog. To do this, you could ask each friend to answer the question “Are you a dog?” in the following way. Each friend should flip a coin in secret, and answer the question truthfully if the coin came up heads; but, if the coin came up tails, that friend should always sa…

The proper way is: you flip a coin, if it comes up heads, you say the truth, otherwise you say whatever you want.

Then you infer an estimate using Bayes' theorem.

Otherwise it is not private, as a reply has pointed out.

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#12

Post by yorwba »

mirimir wrote: post #5
omarforgotpwd wrote: post #3

Earlier quoted context omitted.

Let's say i'm collecting a simple yes / no piece of data such as... did this user open Google Chrome today. Every day my analytics engine sends the data back up: yes, no, no, yes, yes, yes, no. Someone could look at this data and know whether you used Chrome or not on a given day, but when Apple sends the data up it randomly flips the answer for a certain number of data points in such a way that the flipping effects…

OK, thanks :) But still, there are some questions that you'd arguably never want to say "yes" to. Such as, did you visit some verboten site (terrorist, child porn, etc) today? So how can an algorithm "know" which questions it's safe to use differential privacy with, and which it isn't? Or would you argue that it's safe enough to use differential privacy with even such questions?

You're not really saying "yes", you're saying "I either visited this site or my coin came up heads.". Of course this only works if the other person believes that you flipped a coin beforehand, but in this case it's part of the data-collection procedure, so not much trust is necessary.

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#13

Post by LaGrange »

1_player wrote: post #10
quantisan wrote: post #4

Earlier quoted context omitted.

Let’s say you wanted to count how many of your online friends were dogs, while respecting the maxim that, on the Internet, nobody should know you’re a dog. To do this, you could ask each friend to answer the question “Are you a dog?” in the following way. Each friend should flip a coin in secret, and answer the question truthfully if the coin came up heads; but, if the coin came up tails, that friend should always sa…

Is there a name for this algorithm?

The entire approach is called differential security, as in the headline of the thing we're commenting upon ;-)

But, as someone mentioned, if the coin comes tails you should answer with another coin flip, not "yes".

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#14

Post by cm2187 »

quantisan wrote: post #4
mirimir wrote: post #2

Could someone please ELI5 how an "intimate" provider (such as Apple, Google or Microsoft) can collect any data ongoingly without eventual loss of privacy?

Let’s say you wanted to count how many of your online friends were dogs, while respecting the maxim that, on the Internet, nobody should know you’re a dog. To do this, you could ask each friend to answer the question “Are you a dog?” in the following way. Each friend should flip a coin in secret, and answer the question truthfully if the coin came up heads; but, if the coin came up tails, that friend should always sa…

It doesn't really protect privacy, unless your first coin is highly biased toward the random answer. You can still infer that there is a higher likelihood that this individual is a dog. A few 60-70% reliability inferences on various dog related characteristics and you can identify a dog with 95% chance.

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#15

Post by tgragnato »

mirimir wrote: post #2

Could someone please ELI5 how an "intimate" provider (such as Apple, Google or Microsoft) can collect any data ongoingly without eventual loss of privacy?

Anything collected might break privacy, because that information may be something sensitive. Differential privacy relies upon obfuscating reported data in a statistically meaningful way.

Accordingly, you need to analyse aggregate statistics only, add random inaccuracy and apply data binning, anonymise reports. And you need to calibrate the noise. The latter is what the paper seems to be mainly focused on.

> We call for Apple to make its implementation of privacy-preserving algorithms public and to make the rate of privacy loss fully trans-parent and tunable by the user.

"Calibrating Noise to Sensitivity in Private Data Analysis" is about the matter.

- https://www.microsoft.com/en-us/research/publication/calibra...

- https://link.springer.com/content/pdf/10.1007%2F11681878_14....

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#17

Post by tempay »

tkuichooseyou wrote: post #9
quantisan wrote: post #4

Earlier quoted context omitted.

Let’s say you wanted to count how many of your online friends were dogs, while respecting the maxim that, on the Internet, nobody should know you’re a dog. To do this, you could ask each friend to answer the question “Are you a dog?” in the following way. Each friend should flip a coin in secret, and answer the question truthfully if the coin came up heads; but, if the coin came up tails, that friend should always sa…

In that case though, you would know that the "No" friends are definitely not dogs, and the "Yes" friends are possibly a dog, so it seems like the dogs would still not be completely anonymous. Wouldn't the dogs be better off not partaking in the survey and being narrowed down into a group of possible dogs?

The implementation of this should give a random answer when not being truthful.

If the coin comes you heads you answer truthfully. If it comes up tails, you flip the coin again and answer if the yes if the coin is heads and no if the coin is tails. You can then no longer know if anybody is (or is not) a dog.

The probabilities can be adjusted to provide more or less privacy (while making the data less or more useful). For example, if you only answer truthfully 0.1% of the time it would be hard to know anything about anyone, at the cost of knowing the total number of dogs less precisely.

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#18

Post by makomk »

mirimir wrote: post #2

Could someone please ELI5 how an "intimate" provider (such as Apple, Google or Microsoft) can collect any data ongoingly without eventual loss of privacy?

On an ongoing basis? Probably can't. Differential privacy may work OK for one-off measurements, but it's known to leak when doing ongoing measurements of data that's somewhat correlated over time such as emoji use. It's not clear there's any way around this.

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#19

Post by frankmcsherry »

1_player wrote: post #10
quantisan wrote: post #4

Earlier quoted context omitted.

Let’s say you wanted to count how many of your online friends were dogs, while respecting the maxim that, on the Internet, nobody should know you’re a dog. To do this, you could ask each friend to answer the question “Are you a dog?” in the following way. Each friend should flip a coin in secret, and answer the question truthfully if the coin came up heads; but, if the coin came up tails, that friend should always sa…

Is there a name for this algorithm?

The coin flipping approach is called "Randomized Response" and dates back to the 60s.

https://en.wikipedia.org/wiki/Randomized_response

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#20

Post by frankmcsherry »

omarforgotpwd wrote: post #3
mirimir wrote: post #2

Could someone please ELI5 how an "intimate" provider (such as Apple, Google or Microsoft) can collect any data ongoingly without eventual loss of privacy?

Let's say i'm collecting a simple yes / no piece of data such as... did this user open Google Chrome today. Every day my analytics engine sends the data back up: yes, no, no, yes, yes, yes, no. Someone could look at this data and know whether you used Chrome or not on a given day, but when Apple sends the data up it randomly flips the answer for a certain number of data points in such a way that the flipping effects…

> The more changes get made, the more privacy the user has.

This is a minor, pedantic point, but what you really mean to say is "the closer the changes get to 50%, the more privacy the user has". If you change all results, then it is easy to flip them all back.

This distinction trips up several folks, where the research world initially believed (and some in official stats still believe[1]) that a part of privacy is literally not publishing the true answer (e.g. above: literally flip every output).

What you actually want is

    Pr[output | input] ~= Pr[output | input']
which may mean that you should leave things alone, which feels weird but is important.

[1]: Noise addition is a common way to obscure real-valued data, and some official stats bureaus have the ridiculous rule that "you always add noise, and you never add less than X in absolute value", leading to releases where you can be 100% confident that the true value is not in a range around the published number.

Post reply on HN