Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

arxiv.org

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#31

Post by dpriv123 »

pfortuny wrote: post #11
quantisan wrote: post #4

Earlier quoted context omitted.

Let’s say you wanted to count how many of your online friends were dogs, while respecting the maxim that, on the Internet, nobody should know you’re a dog. To do this, you could ask each friend to answer the question “Are you a dog?” in the following way. Each friend should flip a coin in secret, and answer the question truthfully if the coin came up heads; but, if the coin came up tails, that friend should always sa…

The proper way is: you flip a coin, if it comes up heads, you say the truth, otherwise you say whatever you want. Then you infer an estimate using Bayes' theorem. Otherwise it is not private, as a reply has pointed out.

" The proper way is: you flip a coin, if it comes up heads, you say the truth, otherwise you say whatever you want."

Saying "whatever you want" will incur a very large sampling error, especially as the population of those saying whatever increases.

What is needed is a notion of scalable privacy, where as the population of those saying "whatever" increases the privacy strength also increases yet the absolute error remains at worst constant.

https://arxiv.org/abs/1708.01884

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#33

Post by simonh »

So now Apple's privacy system is only stupidly more secure than everyone else's instead of absurdly more secure.

So 16 per day sounds like a lot more than 1 or 2 per day, but what do these numbers mean? Presumably 16 per day is a theoretical maximum if you were to generate every kind of privacy related data ever day. But is 16 really a lot? How high would that have to cumulatively go in order to be useful for extracting reliable info on an individual? Wouldn't the info collected on an individual still have to be associated with them? Frankly I'm not really able to determine any of that from the paper.

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#34

Post by pfortuny »

dpriv123 wrote: post #31
pfortuny wrote: post #11

Earlier quoted context omitted.

The proper way is: you flip a coin, if it comes up heads, you say the truth, otherwise you say whatever you want. Then you infer an estimate using Bayes' theorem. Otherwise it is not private, as a reply has pointed out.

" The proper way is: you flip a coin, if it comes up heads, you say the truth, otherwise you say whatever you want." Saying "whatever you want" will incur a very large sampling error, especially as the population of those saying whatever increases. What is needed is a notion of scalable privacy, where as the population of those saying "whatever" increases the privacy strength also increases yet the absolute error rem…

Mmmhhh... I was trying to ELI5. I understand the sampling error may be large but I cannot see the inherent problem. Could you explain please? (I mean, what do I have to do if I get tails? or do we change the coin).

Honest question, just too lazy to read the manuscript you link...

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#35

Post by geofft »

mirimir wrote: post #5
omarforgotpwd wrote: post #3

Earlier quoted context omitted.

Let's say i'm collecting a simple yes / no piece of data such as... did this user open Google Chrome today. Every day my analytics engine sends the data back up: yes, no, no, yes, yes, yes, no. Someone could look at this data and know whether you used Chrome or not on a given day, but when Apple sends the data up it randomly flips the answer for a certain number of data points in such a way that the flipping effects…

OK, thanks :) But still, there are some questions that you'd arguably never want to say "yes" to. Such as, did you visit some verboten site (terrorist, child porn, etc) today? So how can an algorithm "know" which questions it's safe to use differential privacy with, and which it isn't? Or would you argue that it's safe enough to use differential privacy with even such questions?

Presumably almost nobody visits the verboten sites, and so if you answer "yes", it's far more likely that you answered "yes" because your coin came up tails than because you actually visited it.

Or, in other words, the question is not "Did you visit this site," the question is "Did your coin come up tails or did you visit this site," which is a perfectly safe question to say "yes" to.

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#36

Post by dpriv123 »

pfortuny wrote: post #34
dpriv123 wrote: post #31

Earlier quoted context omitted.

" The proper way is: you flip a coin, if it comes up heads, you say the truth, otherwise you say whatever you want." Saying "whatever you want" will incur a very large sampling error, especially as the population of those saying whatever increases. What is needed is a notion of scalable privacy, where as the population of those saying "whatever" increases the privacy strength also increases yet the absolute error rem…

Mmmhhh... I was trying to ELI5. I understand the sampling error may be large but I cannot see the inherent problem. Could you explain please? (I mean, what do I have to do if I get tails? or do we change the coin). Honest question, just too lazy to read the manuscript you link...

copying verbatim relevant sections

" the estimation error quickly increases with the population size due to the underlying truthful distribution distortion. For example, say we are interested in how many vehicles are at a popular stretch of the highway. Say we configure flip1 = 0.85 and flip2 = 0.3. We query 10,000 vehicles asking for their current location and only 100 vehicles are at the particular area we are interested in (i.e., 1% of the population truthfully responds “Yes"). The standard deviation due to the privacy noise will be 21 which is slightly tolerable. However, a query over one million vehicles (now only 0.01% of the population truthfully responds “Yes") will incur a standard deviation of 212. The estimate of the ground truth (100) will incur a large absolute error when the aggregated privatized responses are two or even three standard deviations (i.e., 95% or 99% of the time) away from the expected value, as the mechanism subtracts only the expected value of the noise."

"In this paper, our goal is to achieve the notion of scalable privacy. That is, as the population increases the privacy should strengthen. Additionally, the absolute error should remain at worst constant. For example, suppose we are interested in understanding a link between eating red meat and heart disease. We start by querying a small population of say 100 and ask “Do you eat red meat and have heart disease?". Suppose 85 truthfully respond “Yes". If we know that someone participated in this particular study, we can reasonably infer they eat red meat and have heart disease regardless the answer. Thus, it is difficult to maintain privacy when the majority of the population truthfully responds “Yes".

Querying a larger and diverse population would protect the data owners that eat red meat and have heart disease. Let’s say we query a population of 100,000 and it turns out that 99.9% of the population is vegetarian. In this case, the vegetarians blend with and provide privacy protection of the red meat eaters. However, we must be careful when performing estimation of a minority population to ensure the sampling error does not destroy the underlying estimate."

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#37

Post by FlyingLawnmower »

frankmcsherry wrote: post #26
bugmen0t wrote: post #25

Funny that almost everyone in this threat seems to "get" Differential Privacy and thinks of it as a good tool. But when it was discussed for Mozilla Firefox everybody was appalled and enraged. (Thread at https://news.ycombinator.com/item?id=15071492 )

I think the big difference there is that Mozilla was proposing opt-out differentially private data collection, whereas I believe Apple has historically been opt-in (with the default being no data collection).

Do you believe an opt-in in necessary with a (properly implemented) differentially private collection mechanism? Just curious about your take on this.

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#38

Post by yorwba »

cm2187 wrote: post #30
frankmcsherry wrote: post #29

Earlier quoted context omitted.

Yes, I think you've misunderstood. The text you've quoted is about how a random subset of the population is already immune to the issue of repeated queries, not that subsampling the population helps in any way. If you don't interrupt the quotation mid-sentence, it reads: > Even so, some randomly-chosen users need not fear such attacks at all: with probability (1/2 f)^h, clients will generate a Permanent randomized re…

Sampling and taking a random subset of the population are synonymous. But my point is precisely that this technique helps with a single question. As soon as you are doing continuous mass collection you don't really get any privacy protection from this technique, and you have to rely on other techniques (encryption, etc).

There are two kinds of sampling involved here: selecting whom to ask a question, and individuals selecting their responses. The random subset of the population is determined by their own choices, that lead them to never say anything useful. An attacker has no influence on this, so if their target is within that group, the attack can't succeed.

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#39

Post by frankmcsherry »

FlyingLawnmower wrote: post #37
frankmcsherry wrote: post #26

Earlier quoted context omitted.

I think the big difference there is that Mozilla was proposing opt-out differentially private data collection, whereas I believe Apple has historically been opt-in (with the default being no data collection).

Do you believe an opt-in in necessary with a (properly implemented) differentially private collection mechanism? Just curious about your take on this.

I think that is the ethical thing to do, yes.

Differential privacy does statistically disclose information about you, and whether the quantitative bound used is up to your standards is a decision you should make before it happens. Given that user-level understanding of DP is so low, I don't think defaulting people in to levels chosen by others is a good idea. I 100% guarantee Mozilla doesn't have the background to make this choice responsibly, and doesn't have near the DP expertise the RAPPOR team has / had (who I also wouldn't trust to chose things for me).

Ideally, the use of differential privacy would make you more willing to opt in (when you have a choice) rather than being a smokescreen for organizations that simply want to harvest more data (which was Mozilla's stated motivation).

Edit: fwiw, there are some cool "recent" versions of differential privacy that let the users control the amount of privacy loss on a user-by-user basis. So, you could start at 0 and dial it up as you feel more comfortable with the tech. This incentivizes organizations to be more transparent with what they do, as it (in principle) increases turnout.

Edit2: For context, Apple's "default" values appear to be (from this paper) epsilon = 16 * days. That means that each day you are active, the posterior probability someone has about any fact about you can increase by a factor of exp(16) ~= 88 million. So, numbers matter and I am (i) glad Apple made it opt-in, (ii) super disappointed they aren't at all transparent about how it works, and (iii) thankful that the paper authors are doing this work.

Re: Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

#40

Post by pfortuny »

dpriv123 wrote: post #36
pfortuny wrote: post #34

Earlier quoted context omitted.

Mmmhhh... I was trying to ELI5. I understand the sampling error may be large but I cannot see the inherent problem. Could you explain please? (I mean, what do I have to do if I get tails? or do we change the coin). Honest question, just too lazy to read the manuscript you link...

copying verbatim relevant sections " the estimation error quickly increases with the population size due to the underlying truthful distribution distortion. For example, say we are interested in how many vehicles are at a popular stretch of the highway. Say we configure flip1 = 0.85 and flip2 = 0.3. We query 10,000 vehicles asking for their current location and only 100 vehicles are at the particular area we are inte…

OK, so the idea is to keep the estimate useful without harming the privacy. Got it.

Thanks.

Post reply on HN