Could not get hold of the paper. Are they doing Gibbs sampling or a semiparametric variant of that ? https://en.wikipedia.org/wiki/Gibbs_sampling Generating tuples(row) by Gibbs sampling will allow generation of samples from the joint distribution. This in turn would preserve all correlations, conditional probabilities etc. This can be done by starting at a original tuple at random and then repeatedly mutating the tu…
I could download it from here: https://dai.lids.mit.edu/wp-content/uploads/2018/03/SDV.pdf Are you facing any trouble while accessing this link?
Artificial data give the same results as real data without compromising privacy
41–48 of 48 posts
Re: Artificial data give the same results as real data without compromising privacy
#42I'm highly dubious of the ability for synthetic data to model accurately datasets without introducing unexpected bias, esp. to account for causality. If you dig through the original paper, the conclusion is on the line with that: “For 7 out of 15 comparisons, we found no significant difference between the accuracy of features developed on the control dataset vs. those developed on some version of the synthesized data…
I'd like to read the paper before drawing such a conclusion. (the link to it seems to be broken) "for 7 out of 15 comparisons, we found no significant difference" could mean all sorts of things. It could mean that 7 comparisons were perfect and 8 were complete garbage, as you suggest. Or it could mean that 7 comparisons were perfect and 8 had differences that were statistically significant, but the magnitudes of the…
Re: Artificial data give the same results as real data without compromising privacy
#43Earlier quoted context omitted.
I'd like to read the paper before drawing such a conclusion. (the link to it seems to be broken) "for 7 out of 15 comparisons, we found no significant difference" could mean all sorts of things. It could mean that 7 comparisons were perfect and 8 were complete garbage, as you suggest. Or it could mean that 7 comparisons were perfect and 8 had differences that were statistically significant, but the magnitudes of the…
You are ignoring the restrictions and regulation that exist around sharing data in lots of financial, government and medical industries. Sometimes, the missed cost of 5 percent accuracy is much less than the inspections, delays and blockages that otherwise would occur if they wanted to use real data.
But yeah, you're right; I was being oversimplistic in just thinking of it in terms of "can have/can't have" and not considering the, "can have, but at too high a cost" angle.
Re: Artificial data give the same results as real data without compromising privacy
#44The claim is too bold and I would reject this paper They should clarify that the data is good enough for linear regression. Not to say there is no difference between real and syn data.
Re: Artificial data give the same results as real data without compromising privacy
#45I'm highly dubious of the ability for synthetic data to model accurately datasets without introducing unexpected bias, esp. to account for causality. If you dig through the original paper, the conclusion is on the line with that: “For 7 out of 15 comparisons, we found no significant difference between the accuracy of features developed on the control dataset vs. those developed on some version of the synthesized data…
Just below what you reproduced, they write: When we examined the confidence intervals for the remaining 8 tests, we found that for half, the mean of accuracies for features written over synthesized data was higher then for those written on the control dataset. In other words, for 4 out of remaining 8 cases, the models on synthetic data performed better.
Re: Artificial data give the same results as real data without compromising privacy
#46Earlier quoted context omitted.
The idea is to sidestep the need to access private information in order for researchers to do their work. So in this case, the artificial data is more useful, since the real data is inaccessible.
Hi, I'm one of the authors of this work. We're very proud that this has attracted so much attention on Hacker News. I'm happy to answer a few questions. We had two requirements for the synthetic data: From the paper, “This synthetic data must meet two requirements: 1. it must somewhat resemble the original data statistically, to ensure realism and keep problems engaging for data scientists. 2. it must also formally a…
Re: Artificial data give the same results as real data without compromising privacy
#47I haven't read the original paper (yet), but something doesn't sit right with the work, if the way it is portrayed is indeed faithful to it and I'm not missing something important. - It looks like the work of the data scientists will be limited to the extent of the modeling already done by recursive conditional parameter aggregation. (edit: So why not just ship that model and adapt it instead of using it to generate…
Just finished the paper, so let me take a stab: Peeling back the mystery a bit, what is happening is: 1. From each child table upwards, model each column as a simple distribution (e.g. Gaussian) and covariance matrix. 2. Given those child table distribution parameters, pass them back as row values to their respective parent tables. What you end up with is a "flattened" version of each parent table that has the inform…
Table modeling: While column distributions are picked using the KS-test, the covariance matrix calculation first normalizes the column distributions. Assuming that is reasonable, there is a claim of "this model contains all the information about the original table in a compact way..", but it doesn't account for possible multi-dimensional relationships in the data. It only looks at a series of projections to 2D. Can a d-dimensional dataset (in practice) be effectively summarized by the set of projections on to the d(d-1)/2 two-dimensional subspaces? That's once kind of summary, but I'm unsure whether that is adequate for practical modeling work, especially if folks try to apply high dimensional techniques (DL?) to this. (edit: I feel reasonably sure it isn't adequate. If a column ends up being bi-modal, for example, even that gets lost in translation in this approach?)
Crowdsourced validations: The synthetic sets were generated for already available public datasets. It isn't clear from the paper how any bias resulting from prior familiarity with the public datasets would be accounted for in the study concluding equivalence.
Privacy claims: This is a bit unclear. The "apply random noise" technique seems to suggest something similar to differential privacy, but makes no mention of it. If not DP, what definition of "privacy" is being used here? (I'm ok that proving their algorithm to be privacy safe according to a chosen definition of privacy may be out of scope of the paper.)
(Edit2: I can't help the feeling I have that this paper is an elaborate April fool's joke released early ;)
Re: Artificial data give the same results as real data without compromising privacy
#48Earlier quoted context omitted.
Just finished the paper, so let me take a stab: Peeling back the mystery a bit, what is happening is: 1. From each child table upwards, model each column as a simple distribution (e.g. Gaussian) and covariance matrix. 2. Given those child table distribution parameters, pass them back as row values to their respective parent tables. What you end up with is a "flattened" version of each parent table that has the inform…
(Going through the paper .. a few questions/notes) Table modeling: While column distributions are picked using the KS-test, the covariance matrix calculation first normalizes the column distributions. Assuming that is reasonable, there is a claim of "this model contains all the information about the original table in a compact way..", but it doesn't account for possible multi-dimensional relationships in the data. It…
Just thinking out loud here:
The typical case where a low dimensional representation would fail you is if you had dependencies (e.g. bimodal relations) that weren't represented by a datatype or foreign key. Recall that the simulation of data still occurs within each table, so the higher the non-represented inter-table dimensionality is, the supplied distributions can measure it. It's might be that, for the most part, the raw columns (not from child tables) have much more bearing on the merit of the table covariance. This seems natural, due to the semantic nature of RDBMS structures.
It's probably an important caveat that typical RDBMS structures are created to optimize the user's understanding of the data through semantic structure. Since the claim of the paper was only that they could provide a useful abstraction for simulation, I think it's OK to proceed with the assumption that Gaussians can never be fully sufficient in modeling highly dimensional data without help.
There are existing non-parametric models that attempt to do a similar thing for relational data that I think are more promising. One drawback of current solutions like BayesDB is that you're still dealing with the original table structure, which this paper tries to get around. It would be nice to bridge the gap for something like PyMC3 where we find a cute way to flatten the data, like this paper.
[1] Probabilistic Search for Structured Data via Probabilistic Programming and Nonparametric Bayes. https://arxiv.org/pdf/1704.01087.pdf