Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How to run analytics on data without access to the data?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 46 posts

Re: Ask HN: How to run analytics on data without access to the data?

#31

Post by dumbfounder »

franky47 wrote: post #10
dumbfounder wrote: post #6

Earlier quoted context omitted.

Client side is the first answer, but is there a second? Is there a way to peer review a piece of code that can run in a 3rd party container (peer review and cryptographically signed), such that the actual container running the code is encrypted itself and can run anywhere? I am imagining you download the "container", put the data in, encrypt the container with the data inside, and have that run anywhere. But I have n…

That would be feasible with homomorphic encryption, however current implementations are very far from practical applications (extreme resource consumption, terrible performance).

I am not talking about just encrypting the data and performing computation on the encrypted data, but encrypting the entire container with data inside and running that to produce a result with no way to view what’s going on inside. You can get around the limitations with how to run an algorithm on encrypted data because the data is not encrypted with respect to the program itself.

Theoretically it would work like this: you download a docker image, you load your data into it, you encrypt the entire image with data inside, you send that whole package to the cloud where it is run and it produces an output.

Re: Ask HN: How to run analytics on data without access to the data?

#32

Post by lmkg »

brian_spiering wrote: post #23

Differential privacy is the field of study for sharing sensitive data in a way that allows analysis while retaining some guarantees of privacy.

Agreed, Differential Privacy is the name for this problem.

Quick summary of important results: You will always leak a small amount of information. But it is possible to bound this leak to whatever level you consider "acceptable." The trade-off is statistical validity of the results (the usual approach adds "noise" to the data and/or analysis).

Re: Ask HN: How to run analytics on data without access to the data?

#33

Post by syats »

I'll tell you about International Data Spaces Assocation, just for the sake of completeness, and because others have mentioned some sort of certification of apps, etc. Finding a general solution to the problem posed by OP is quite difficult, as it requires a lot of extra infrastructure, technical and non-technical.

One idea would be:

1. distribute to the data owners a base system (something that can "run" stuff on their premises). People here have mentioned browsers, but for a more intensive processing this might not be enough.. so think of a docker daemon, keys for some docker registries, etc.

2. have a trusted "app store" (e.g. a docker registry where images are built in a reproducible manner from code which is inspected and certified, and then are cryptographically signed)

3. make a well described interface to the apps to consume the data (thinking of the general use case here.. if you just want to analyze fb info then you can make an adhoc parser...)

4. Have the data owner download, check the signature of, configure and run the app on their premises.

Things get even more interesting when the analytics need data from different non-trusting partners, so that Homeomorphic Encryption becomes necessary.

There is at least one specification that aims at supporting all of this: https://www.internationaldataspaces.org/wp-content/uploads/2... although implementation is, so far, lagging behind.

Re: Ask HN: How to run analytics on data without access to the data?

#34

Post by cipherboy »

michealr wrote: post #26
cipherboy wrote: post #20

Earlier quoted context omitted.

I'm a little rusty but I swear I saw a partial homomorphic encryption scheme for aggregates and analytics. I want to say Enigma conference, '16 or '17? Maybe by Boston University. The benefit being that while you can run any computatio with a FHE, PHEs are generally faster. IIRC Microsoft was also doing research on PHEs.

Interesting, did Microsoft bring the research any further?

https://dl.acm.org/doi/abs/10.1145/2463676.2467797

Might be what I was thinking of. I'm sure you can find other publications as well. I'm no longer at University and I've lost touch with that professor so I'm not sure their current research.

Re: Ask HN: How to run analytics on data without access to the data?

#35

Post by stelfer »

Take a look at Google Private Join and Compute[1]. But be aware that the problem you frame is an unsolved research problem with an active global community. The topics you are looking for are applications of secure multiparty computation and homomorphic encryption. Also, be ready for something as simple as a column join to take 24 hours per query.

[1] https://github.com/Google/private-join-and-compute

Re: Ask HN: How to run analytics on data without access to the data?

#36

Post by franky47 »

dumbfounder wrote: post #31
franky47 wrote: post #10

Earlier quoted context omitted.

That would be feasible with homomorphic encryption, however current implementations are very far from practical applications (extreme resource consumption, terrible performance).

I am not talking about just encrypting the data and performing computation on the encrypted data, but encrypting the entire container with data inside and running that to produce a result with no way to view what’s going on inside. You can get around the limitations with how to run an algorithm on encrypted data because the data is not encrypted with respect to the program itself. Theoretically it would work like thi…

Yes I thought of that for my analytics SaaS (to calculate weekly reports), the issue is that the image has to be decrypted on the Docker host before execution, which requires it having access to the key somehow, breaking the end-to-end encryption promise (ie: "we have no way to access your data").

This could be mitigated by having that worker host self-hosted by your clients, it depends how practical that might be.

Re: Ask HN: How to run analytics on data without access to the data?

#37

Post by satyrnein »

meowface wrote: post #15

Your best bet is probably to just do all the processing locally in the browser. The issue is 1) from most end users' perspectives, they have no idea if it's actually running locally or talking to a server, or how to verify it, or probably what that difference even means in the first place, so a skeptical user won't necessarily gain that much additional peace of mind, and 2) hypothetically a compromise could still res…

Maybe a browser extension with limited permissions? Say the tool looks at Slack and counts how often you use the ROFL emoji. The extension could be granted access to *.slack.com but no other domains.

Re: Ask HN: How to run analytics on data without access to the data?

#38

Post by meowface »

satyrnein wrote: post #37
meowface wrote: post #15

Your best bet is probably to just do all the processing locally in the browser. The issue is 1) from most end users' perspectives, they have no idea if it's actually running locally or talking to a server, or how to verify it, or probably what that difference even means in the first place, so a skeptical user won't necessarily gain that much additional peace of mind, and 2) hypothetically a compromise could still res…

Maybe a browser extension with limited permissions? Say the tool looks at Slack and counts how often you use the ROFL emoji. The extension could be granted access to *.slack.com but no other domains.

That could work, but then you have the additional barrier of having to convince people to install your browser extension, and for people who are already worried about privacy, that comes with its own can of worms. Especially if they don't necessarily understand or trust the permission model.

Re: Ask HN: How to run analytics on data without access to the data?

#39

Post by meowface »

michealr wrote: post #22
hunter2_ wrote: post #19

Earlier quoted context omitted.

This could work if the analytics engine is free and (ported to) JavaScript, but not if it's closed source. In the latter case, a trusted third party (escrow, one might call it) as OP described does seem like the way to go. The problem is, why would end users trust the third party more than the analytics developer? Are there companies that specialize in being this third party and have amassed mutual trust of the gener…

Analytics wise, I'm ok with being restricted, other commenters have mentioned looking at WASM as a possible workaround. So local does seem to make the most sense, practicality wise A thought, the possible scope of services in the data notary or data escrow side of things does seem like an underexplored product category.

Any such data notary/escrow company has a pretty good shot of eventually getting breached (they'd naturally be a prime target, since the attackers could get tons of data from tons of people on behalf of tons of different companies), and that'll possibly destroy that company and maybe also your app. There's also the risk they may eventually have rogue employees, etc.

Re: Ask HN: How to run analytics on data without access to the data?

#40

Post by Syzygies »

https://mathscinet.ams.org/mathscinet/help/about.html "MathSciNet® is an electronic publication offering access to a carefully maintained and easily searchable database of reviews, abstracts and bibliographic information for much of the mathematical sciences literature. Over 125,000 new items are added each year..."

The stakes are lower when money, not privacy, is at risk. I have attempted to argue for years that the MathSciNet catalog of the mathematical literature should be open to all forms of machine learning and mind mapping software experiments. It remains a cash cow for the American Mathematical Society, and they're fiercely proud of its human curation by 19th century methods. Meanwhile, mathematicians continue to believe that math remains separated into tribes, with number theorists lobbying to hire their own at departmental meetings. The true connections between ideas defy these ancient categories. I see a generation of potential advances squandered by not letting third-party tools in to study MathSciNet.

The right ideas could help here. One isn't protecting individual privacy, just a cash cow. The bar is lower.

Post reply on HN