Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How to run analytics on data without access to the data?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 46 posts

Ask HN: How to run analytics on data without access to the data?

#1

Post by michealr »

I have little service for personal use, and I was considering opening it up to a general audience. Right now its processing some of my personal data for fun little personal report, in particular chat data. Since its information I have access to already, I don't mind running the program locally. What I would like to be able to do is run an analysis for anyone and return the little report that I get for myself. Without having access to their data or storing it in the first place. I know with for example oauth scopes, you can grant access, which sort of fits the criteria. But I'm thinking more exported data from an application, that doesn't have delegated access functionality

How I envisioned a solution would be some trusted third party takes my analysis script, returns the report and that is it. I never see the underlying data and recieve only one time token to access it.

I know it will never be hundred percent leak proof, and there is still a level of user trust, I realise that, but just thinking conceptually, is there any existing service out there, that does such a thing or attempts to offer something similar? Or what would an alternative approach look like?

Re: Ask HN: How to run analytics on data without access to the data?

#2

Post by gostsamo »

Adding the third party only complicates the issue because the user will have to trust you and the proxy, and the proxy will have to trust your code. Best case, let the user download your code as a mobile or desktop app and run the analysis themselves.

Re: Ask HN: How to run analytics on data without access to the data?

#3

Post by franky47 »

I asked myself a similar question for web analytics a year ago [1]: how to provide a service without having access to the underlying data. It requires shifting the processing onto the client side, so it limits what you can do, but it's best for privacy, and security (since the data never leaves the native app or browser).

[1] https://chiffre.io

Re: Ask HN: How to run analytics on data without access to the data?

#4

Post by BelenusMordred »

> I know it will never be hundred percent leak proof

A slow leaking ship will still sink. Attempts so far to anonymise public datasets have been terrible and turned into a garbage fire by attackers every time with minimal effort. Don't hand out false promises.

Guess you are looking for fully homomorphic encryption. A long-outstanding problem with lots of smart people working on it, some are doing ok at getting there.

https://github.com/ibm/fhe-toolkit-linux

Re: Ask HN: How to run analytics on data without access to the data?

#6

Post by dumbfounder »

franky47 wrote: post #3

I asked myself a similar question for web analytics a year ago [1]: how to provide a service without having access to the underlying data. It requires shifting the processing onto the client side, so it limits what you can do, but it's best for privacy, and security (since the data never leaves the native app or browser). [1] https://chiffre.io

Client side is the first answer, but is there a second? Is there a way to peer review a piece of code that can run in a 3rd party container (peer review and cryptographically signed), such that the actual container running the code is encrypted itself and can run anywhere?

I am imagining you download the "container", put the data in, encrypt the container with the data inside, and have that run anywhere.

But I have no idea if that is possible.

Re: Ask HN: How to run analytics on data without access to the data?

#7

Post by michealr »

BelenusMordred wrote: post #4

> I know it will never be hundred percent leak proof A slow leaking ship will still sink. Attempts so far to anonymise public datasets have been terrible and turned into a garbage fire by attackers every time with minimal effort. Don't hand out false promises. Guess you are looking for fully homomorphic encryption. A long-outstanding problem with lots of smart people working on it, some are doing ok at getting there.…

Thats true, a poorly chosen description on my part.

Very cool, had read about homomorphic systems. For fully homomorphic systems has there been successful SAAS like offering allowing use of such a systems? Or do you think its still in the research oriented phase?

Re: Ask HN: How to run analytics on data without access to the data?

#8

Post by jedimastert »

Either the first party (i.e. the client) runs the data on their own turf or they hand the data to someone else (you or whatever third-party you use) and trust that the other end is going to treat your data right.

I'm sure there's some sort of homomorphic encryption[0] magic scheme that might let you process the data on other servers or something, but I could not even begin to tell you how. Really, it's just trust.

Re: Ask HN: How to run analytics on data without access to the data?

#9

Post by michealr »

dumbfounder wrote: post #6
franky47 wrote: post #3

I asked myself a similar question for web analytics a year ago [1]: how to provide a service without having access to the underlying data. It requires shifting the processing onto the client side, so it limits what you can do, but it's best for privacy, and security (since the data never leaves the native app or browser). [1] https://chiffre.io

Client side is the first answer, but is there a second? Is there a way to peer review a piece of code that can run in a 3rd party container (peer review and cryptographically signed), such that the actual container running the code is encrypted itself and can run anywhere? I am imagining you download the "container", put the data in, encrypt the container with the data inside, and have that run anywhere. But I have n…

I wonder myself the same thing.

Thinking through issues, the external script could still repeatable run on the hidden data, slowly building an idea of the information. There are techniques like homomorphic encryption that go in the direction of allowing analysis on encrypted data.

Musing on possible other solutions, I wonder if simply ratching up the cost and repeated access and limiting data output would discourage this profile building.

Another possibility is it possible to concieve of the service, that takes in a script, runs it, and then tests the returned data for the level of information entropy. Blocking anything above a certain threshold. FYI not sure if that is complete nonsense, but conceptually, with much hand waving, maybe it works.

Going local though does help too

Re: Ask HN: How to run analytics on data without access to the data?

#10

Post by franky47 »

dumbfounder wrote: post #6
franky47 wrote: post #3

I asked myself a similar question for web analytics a year ago [1]: how to provide a service without having access to the underlying data. It requires shifting the processing onto the client side, so it limits what you can do, but it's best for privacy, and security (since the data never leaves the native app or browser). [1] https://chiffre.io

Client side is the first answer, but is there a second? Is there a way to peer review a piece of code that can run in a 3rd party container (peer review and cryptographically signed), such that the actual container running the code is encrypted itself and can run anywhere? I am imagining you download the "container", put the data in, encrypt the container with the data inside, and have that run anywhere. But I have n…

That would be feasible with homomorphic encryption, however current implementations are very far from practical applications (extreme resource consumption, terrible performance).
Post reply on HN