What about Fully Homomorphic encryption? Would a FHE scheme enable to discover patterns without seeing the data?
Ask HN: How to run analytics on data without access to the data?
11–20 of 46 posts
Re: Ask HN: How to run analytics on data without access to the data?
#12I asked myself a similar question for web analytics a year ago [1]: how to provide a service without having access to the underlying data. It requires shifting the processing onto the client side, so it limits what you can do, but it's best for privacy, and security (since the data never leaves the native app or browser). [1] https://chiffre.io
Doable if you think client side of Ethereum or IPFS
Re: Ask HN: How to run analytics on data without access to the data?
#13How is the service written? I'd look to compile it down to WASM or otherwise run it in the browser, if possible.
Client side does make sense.
I guess the user could upload there chat data zip file to the client side app. Which then locally would do the processing. The report itself could be saved, but not the data.
Re: Ask HN: How to run analytics on data without access to the data?
#14What about Fully Homomorphic encryption? Would a FHE scheme enable to discover patterns without seeing the data?
I theory I assume it would, my bottleneck would be just knowledge. Just don't know enough about FHE to comfortably work with it. FHE as a service would be my little mini dream.
Re: Ask HN: How to run analytics on data without access to the data?
#15Your best bet is probably to just do all the processing locally in the browser. The issue is 1) from most end users' perspectives, they have no idea if it's actually running locally or talking to a server, or how to verify it, or probably what that difference even means in the first place, so a skeptical user won't necessarily gain that much additional peace of mind, and 2) hypothetically a compromise could still result in the local data being siphoned off by an attacker. The latter's still a risk for regular desktop applications, but a bit less so (since you can get a signed binary).
The homomorphic encryption approach probably isn't worth the effort. There's always going to be a trade-off between doing something useful and sufficiently/securely obfuscating/anonymizing the data. So I'd recommend the local approach, with a prominent explanation of how you don't and can't see any of the data.
Re: Ask HN: How to run analytics on data without access to the data?
#16> I know it will never be hundred percent leak proof A slow leaking ship will still sink. Attempts so far to anonymise public datasets have been terrible and turned into a garbage fire by attackers every time with minimal effort. Don't hand out false promises. Guess you are looking for fully homomorphic encryption. A long-outstanding problem with lots of smart people working on it, some are doing ok at getting there.…
Thats true, a poorly chosen description on my part. Very cool, had read about homomorphic systems. For fully homomorphic systems has there been successful SAAS like offering allowing use of such a systems? Or do you think its still in the research oriented phase?
Not really SaaS, and debatable whether it's successful, but I hear numerai uses homomorphic encryption when providing data for quants to backtest.
https://numer.ai/ https://en.wikipedia.org/wiki/Numerai
EDIT: added qualifier, since i do not know for sure if numerai is using homomorphic encryption.
Re: Ask HN: How to run analytics on data without access to the data?
#17Could you send your code to their execution environment for a one time run (unlocked with a code?)
Re: Ask HN: How to run analytics on data without access to the data?
#18> I know it will never be hundred percent leak proof A slow leaking ship will still sink. Attempts so far to anonymise public datasets have been terrible and turned into a garbage fire by attackers every time with minimal effort. Don't hand out false promises. Guess you are looking for fully homomorphic encryption. A long-outstanding problem with lots of smart people working on it, some are doing ok at getting there.…
Thats true, a poorly chosen description on my part. Very cool, had read about homomorphic systems. For fully homomorphic systems has there been successful SAAS like offering allowing use of such a systems? Or do you think its still in the research oriented phase?
Definitely still in the research phase for what you are looking for. Performance is anywhere from 5-15 magnitudes slower than ordinary computations. Yes, magnitudes...
Re: Ask HN: How to run analytics on data without access to the data?
#19Your best bet is probably to just do all the processing locally in the browser. The issue is 1) from most end users' perspectives, they have no idea if it's actually running locally or talking to a server, or how to verify it, or probably what that difference even means in the first place, so a skeptical user won't necessarily gain that much additional peace of mind, and 2) hypothetically a compromise could still res…
This could work if the analytics engine is free and (ported to) JavaScript, but not if it's closed source. In the latter case, a trusted third party (escrow, one might call it) as OP described does seem like the way to go.
The problem is, why would end users trust the third party more than the analytics developer? Are there companies that specialize in being this third party and have amassed mutual trust of the general public (akin to a notary public) for handling data and code without leaking either?
Re: Ask HN: How to run analytics on data without access to the data?
#20> I know it will never be hundred percent leak proof A slow leaking ship will still sink. Attempts so far to anonymise public datasets have been terrible and turned into a garbage fire by attackers every time with minimal effort. Don't hand out false promises. Guess you are looking for fully homomorphic encryption. A long-outstanding problem with lots of smart people working on it, some are doing ok at getting there.…
I'm a little rusty but I swear I saw a partial homomorphic encryption scheme for aggregates and analytics. I want to say Enigma conference, '16 or '17? Maybe by Boston University.
The benefit being that while you can run any computatio with a FHE, PHEs are generally faster.
IIRC Microsoft was also doing research on PHEs.