Live data from Hacker News

Ask HN: How to run analytics on data without access to the data?

news.ycombinator.com

21–30 of 46 posts

Re: Ask HN: How to run analytics on data without access to the data?

#21
We [0] are getting quite far decomposing algorithms symbolically and then doing some fancy footwork with private set intersection. It ends up being better/faster/cheaper than homomorphic in a lot of use cases.

Shoot us a note -- would love to hear more details.

[0]: https://proofzero.io

Re: Ask HN: How to run analytics on data without access to the data?

#22

Your best bet is probably to just do all the processing locally in the browser. The issue is 1) from most end users' perspectives, they have no idea if it's actually running locally or talking to a server, or how to verify it, or probably what that difference even means in the first place, so a skeptical user won't necessarily gain that much additional peace of mind, and 2) hypothetically a compromise could still res…

This could work if the analytics engine is free and (ported to) JavaScript, but not if it's closed source. In the latter case, a trusted third party (escrow, one might call it) as OP described does seem like the way to go. The problem is, why would end users trust the third party more than the analytics developer? Are there companies that specialize in being this third party and have amassed mutual trust of the gener…

Analytics wise, I'm ok with being restricted, other commenters have mentioned looking at WASM as a possible workaround. So local does seem to make the most sense, practicality wise

A thought, the possible scope of services in the data notary or data escrow side of things does seem like an underexplored product category.

Re: Ask HN: How to run analytics on data without access to the data?

#24
There's another approach you can do -- make the analysis portable instead.

Assuming data is in a standard format then you can share your script for people to run themselves. Obviously this is fairly difficult in practice unless you can bundle everything into a client-side script on a website.

For reference Narrator [1] does this -- it puts data into a standard format so that analyses written for one company can be run for another. I'm not suggesting you build your stuff on that platform, but it's an interesting approach that does exist.

[1] https://www.narrator.ai

Re: Ask HN: How to run analytics on data without access to the data?

#25

> I know it will never be hundred percent leak proof A slow leaking ship will still sink. Attempts so far to anonymise public datasets have been terrible and turned into a garbage fire by attackers every time with minimal effort. Don't hand out false promises. Guess you are looking for fully homomorphic encryption. A long-outstanding problem with lots of smart people working on it, some are doing ok at getting there.…

Differential privacy is an area that makes some guarantees about not letting personal information about individuals escape. Might be a useful technique as well.

https://en.wikipedia.org/wiki/Differential_privacy

Agree that strong guarantees about privacy aren't achievable.

Re: Ask HN: How to run analytics on data without access to the data?

#26

> I know it will never be hundred percent leak proof A slow leaking ship will still sink. Attempts so far to anonymise public datasets have been terrible and turned into a garbage fire by attackers every time with minimal effort. Don't hand out false promises. Guess you are looking for fully homomorphic encryption. A long-outstanding problem with lots of smart people working on it, some are doing ok at getting there.…

I'm a little rusty but I swear I saw a partial homomorphic encryption scheme for aggregates and analytics. I want to say Enigma conference, '16 or '17? Maybe by Boston University. The benefit being that while you can run any computatio with a FHE, PHEs are generally faster. IIRC Microsoft was also doing research on PHEs.

Interesting, did Microsoft bring the research any further?

Re: Ask HN: How to run analytics on data without access to the data?

#27

Your best bet is probably to just do all the processing locally in the browser. The issue is 1) from most end users' perspectives, they have no idea if it's actually running locally or talking to a server, or how to verify it, or probably what that difference even means in the first place, so a skeptical user won't necessarily gain that much additional peace of mind, and 2) hypothetically a compromise could still res…

This could work if the analytics engine is free and (ported to) JavaScript, but not if it's closed source. In the latter case, a trusted third party (escrow, one might call it) as OP described does seem like the way to go. The problem is, why would end users trust the third party more than the analytics developer? Are there companies that specialize in being this third party and have amassed mutual trust of the gener…

It doesn't need to be in JavaScript. Any language that can compile to WebAssembly would work too. But I agree with the broader point - the code needs to execute on the client, not the server.

Re: Ask HN: How to run analytics on data without access to the data?

#29
This reminds me of https://opensafely.org/, which analyses NHS medical data for research purposes by asking Doctors to run queries on their patient databases and send back only summaries, e.g. "How many of your patients with HIV also had Covid19" https://github.com/opensafely/hiv-research

Re: Ask HN: How to run analytics on data without access to the data?

#30

> I know it will never be hundred percent leak proof A slow leaking ship will still sink. Attempts so far to anonymise public datasets have been terrible and turned into a garbage fire by attackers every time with minimal effort. Don't hand out false promises. Guess you are looking for fully homomorphic encryption. A long-outstanding problem with lots of smart people working on it, some are doing ok at getting there.…

Differential privacy is an area that makes some guarantees about not letting personal information about individuals escape. Might be a useful technique as well. https://en.wikipedia.org/wiki/Differential_privacy Agree that strong guarantees about privacy aren't achievable.

Yes, I found this the other day: https://smartnoise.org/

Need to read more about the concept. Anyone with more good resources?

Post reply on HN