Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Sarus (YC W22) – Work on sensitive data with differential privacy

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 50 posts

Re: Launch HN: Sarus (YC W22) – Work on sensitive data with differential privacy

#21

Post by maximeago »

ZeroCool2u wrote: post #13

So, I work in an org that has truly sensitive data and this has been a barrier for us more times than I can count, so this is obviously very interesting to us and something we've thought about a lot. A couple questions I have are: 1. How well does Sarus work with data that is not in a database, like unstructured data such as documents/text? 2. How does Sarus handle 'legacy' DB's, where the schema for a table might no…

1. Sarus works on data that is organized in records. The intuition is that one record should not transpire in the results (hence protecting their privacy) but studying all records conjointly should be possible. It may be flat files, parquet filets, etc. but we do need this record-level organization. In a given record, there may be columns that are text or images, Sarus will work fine. We never worked on pdf documents. Conceptually it could work but this is quite far down the road.

2. Sarus has connectors to the main DB and we add more when we meet them. The basic assumption is that the experience should be the same as working on the data in its original form. For instance if your data is in a CSV with a weird date format, you will be able to (i) get synthetic data with this same weird date format, (ii) apply python code that transforms this weird date format into something more conventional and use that reformatted version. When running your data job, Sarus will apply your preprocessing code and take it from there.

3. Today we have a python SDK and a SQL connector. Both leverage the same low-level API. We may build other SDKs for other languages but haven't started doing so.

4. Indeed, we don't have any cert yet but we are looking into getting some soon. We are about to start Soc2 for instance. This is somewhat less of a requirement as we never host any of our clients' data. Of course, everything that helps get the green light of the ITSec team is useful.

5. The python SDK is standard python code so you can use in any python env. The notebook is just here to make it more user-friendly in demos. Same for SQL, you can use any SQL querying tool, we did the demo with Metabase.

6. The easiest way is to deploy a docker image with Docker compose. It does not scale on multiple machine yet (stay tuned). In that sense, big data sources are only partially supported: if the source is RedShift and you submit a SQL query to the API, we'll rewrite it and send it to Redshift (which scales), but if you want to do ML on the same data, we won't be able to scale the same.

7. Complex time series is not a problem for the remote execution part provided it is stored in a traditional format. That being said, we don't have a specific synthetic data model for time series yet, so that part of the experience will be a bit different.

8. This is a debate we leave to researchers because there is not a single answer. It depends directly on the number of records in your dataset and the dimensionality of your data. However, you can set up privacy policies so that the weights of ML model without DP are allowed to be shared. This is considered acceptable by 99% of compliance teams in the world today so it's not a huge compromise. If you use Sarus this way, you are guaranteed to have exactly the same performance.

Would love to continue the conversation offline of course!

Re: Launch HN: Sarus (YC W22) – Work on sensitive data with differential privacy

#22

Post by maximeago »

amzl wrote: post #15

Pretty impressive. What do you use for synthetic data generation? Also, you say in the blog post that it works with any type of data. Can you tell a bit more? Does it work for text and images?

We developed our own generative model for synthetic data generation. It is an autoregressive model where each variable is derived from previously generated ones using Transformers networks. If you are interested, you have more details in: https://arxiv.org/pdf/2202.02145.pdf When we say it works on any types of data, we mean: numerical, categorical, text, images and compositions of those types (see the paper).

Re: Launch HN: Sarus (YC W22) – Work on sensitive data with differential privacy

#23

Post by XCSme »

It's unclear for me from the landing page hero section what the product is/does or what problem does it solve:

"PRIVACY-BY-DESIGN

Time-to-data: from months to minutes

Organizations that use Sarus outperform their peers at execution speed for machine learning and analytics while being more secure "

Re: Launch HN: Sarus (YC W22) – Work on sensitive data with differential privacy

#24

Post by maximeago »

fluidcruft wrote: post #20
maximeago wrote: post #16

Earlier quoted context omitted.

Sarus is designed for all data use cases, provided that access to a given user's information is not the objective. This is the case for all of BI, analytics, or machine learning. It also works for testing or debugging, building APIs, etc. It resonates with organizations' aspiration for the democratization of data. Differential privacy provides much better protection than data masking, but most importantly, it does no…

I think this is interesting but I'm having trouble seeing how it would apply to the sorts of machine learning tasks that are drawing heavy interest in a radiology department. How does it apply to, say, development or testing of image segmentation tools? Quite often vendors want to sell us software and we would very much like to test it at scale on our own data to see whether it's trash or not because procurement is a…

Here is how it would work in theory (not including the scalability question of working with heavy DICOM files and huge DNN). I'm assuming your data is made of records composed by an image and some information about the image or the patient.

The system will generate a fake dataset with the exact same structure and schema (the information on patients is realistic, the images look reasonable and importantly has the right encoding, size, etc.). The purpose of this fake data is for the vendor to adjust their algorithm to be able to consume your data as it is. The vendor builds up the preprocessing on the fake data and then submit their data job to the API (say a preprocessing function to be applied on each record and a Tensorflow model to be fitted on the data, or just to measure the performance on the data). The preprocessing code runs on the original records, the model would be trained or validated against the real data. In the end they can prove the value of their model without having to get their hands on the real data.

Re: Launch HN: Sarus (YC W22) – Work on sensitive data with differential privacy

#25

Post by maximeago »

XCSme wrote: post #23

It's unclear for me from the landing page hero section what the product is/does or what problem does it solve: "PRIVACY-BY-DESIGN Time-to-data: from months to minutes Organizations that use Sarus outperform their peers at execution speed for machine learning and analytics while being more secure "

The product solves the problem of the time it takes to access sensitive data for analytics and machine learning. When you work in a large healthcare or financial organization, each dataset is highly protected. Each time a data practitioner needs to work on it, they may have to wait for months for compliance processes to opine on a data masking strategy and engineering teams to prepare a data lab and implement this strategy. With Sarus, data practitioner no longer need to access data to do analytics or machine learning on sensitive data assets.

When internal access to personal data is not a concern within an organization, data sharing with external partners certainly is. This process can be avoided just the same.

Hence the promise of taking time-to-data form months to minutes.

Hope that helps clarify.

Re: Launch HN: Sarus (YC W22) – Work on sensitive data with differential privacy

#26

Post by maximeago »

leecarraher wrote: post #5

for synthetic data generation, what methods are they using to sample data from the distribution? What assumptions about the distribution are being made? Does it model correlations between sample attributes that could adversely effect some ML methods (multi-colinearity can cause problems).

We developed our own generative model for synthetic data generation. It is an autoregressive model where each variable/attribute is derived from previously generated ones using Transformers networks (more details there: https://arxiv.org/pdf/2202.02145.pdf). So yes, correlations are modelled, although exact multicollinearity (when there is a linear relationship between bunch of attributes) would be a bit blurry in the synthetic data.

This being said, the goal of Sarus is to enable analysis on the original data with privacy guarantee on the result (synthetic data is merely used as a tool and a fallback when there is no better solution) so you can write a statistical test to detect multicollinearity and run it on the original data within Sarus.

Re: Launch HN: Sarus (YC W22) – Work on sensitive data with differential privacy

#27

Post by XCSme »

maximeago wrote: post #25
XCSme wrote: post #23

It's unclear for me from the landing page hero section what the product is/does or what problem does it solve: "PRIVACY-BY-DESIGN Time-to-data: from months to minutes Organizations that use Sarus outperform their peers at execution speed for machine learning and analytics while being more secure "

The product solves the problem of the time it takes to access sensitive data for analytics and machine learning. When you work in a large healthcare or financial organization, each dataset is highly protected. Each time a data practitioner needs to work on it, they may have to wait for months for compliance processes to opine on a data masking strategy and engineering teams to prepare a data lab and implement this st…

Thanks, this makes it a lot more clear!

Maybe the hero text could be more clear, explaining in summary what it does (similar to this comment). "Get instant access to sensitive data for analytics and machine learning."

Re: Launch HN: Sarus (YC W22) – Work on sensitive data with differential privacy

#28

Post by fluidcruft »

maximeago wrote: post #24
fluidcruft wrote: post #20

Earlier quoted context omitted.

I think this is interesting but I'm having trouble seeing how it would apply to the sorts of machine learning tasks that are drawing heavy interest in a radiology department. How does it apply to, say, development or testing of image segmentation tools? Quite often vendors want to sell us software and we would very much like to test it at scale on our own data to see whether it's trash or not because procurement is a…

Here is how it would work in theory (not including the scalability question of working with heavy DICOM files and huge DNN). I'm assuming your data is made of records composed by an image and some information about the image or the patient. The system will generate a fake dataset with the exact same structure and schema (the information on patients is realistic, the images look reasonable and importantly has the righ…

The problem we generally have is that plugging the vendor's [insert tensorflow model component] into our network seems to always become an operational no-go prior to purchase due to a variety of reasons including intrusiveness and questions about privacy and the vendor's ability to manipulate the process to get access to datasets. So it's actually the preprocessing step that's we keep hitting as the pain point. In some cases we generate de-identified datasets for demonstration and testing but it can be very labor intensive.

I've not encountered differential privacy in my work before now, but at least for dealing with metadata in the DICOM it could probably be helpful for some datasets. But it could still be challenging to ensure the IODs are correct (or that known quirks are preserved). Anyway this is very interesting. I have a colleague who is working on some utilization/value research using billing records and I'll show him this.

Re: Launch HN: Sarus (YC W22) – Work on sensitive data with differential privacy

#29

Post by maximeago »

fluidcruft wrote: post #28
maximeago wrote: post #24

Earlier quoted context omitted.

Here is how it would work in theory (not including the scalability question of working with heavy DICOM files and huge DNN). I'm assuming your data is made of records composed by an image and some information about the image or the patient. The system will generate a fake dataset with the exact same structure and schema (the information on patients is realistic, the images look reasonable and importantly has the righ…

The problem we generally have is that plugging the vendor's [insert tensorflow model component] into our network seems to always become an operational no-go prior to purchase due to a variety of reasons including intrusiveness and questions about privacy and the vendor's ability to manipulate the process to get access to datasets. So it's actually the preprocessing step that's we keep hitting as the pain point. In so…

Thanks! Our goal is that no matter what preprocessing function they pass, the only end up accessing outputs that comply with the privacy policies. The code gets access to the real data but it is shielded from the vendor who can only see protected outputs. It should address the risk of private information being exposed to them, but for sure, the more sophisticated the preprocessing code will be, the more challenging it will become. Deep learning on Dicom data is pushing the system to the edge a bit.

Re: Launch HN: Sarus (YC W22) – Work on sensitive data with differential privacy

#30

Post by maximeago »

amitport wrote: post #19
maximeago wrote: post #3

Earlier quoted context omitted.

The first link is the corporate website, it may not include all the product details you expected, sorry about that. You should get a lot more details on how it works if you try the tutorial and play with it yourself. This is at the bottom of the post, hopefully it satisfies your curiosity but happy to answer outstanding questions here of course.

Is this a productized Duet[1]? Are you using it under the hood? (As far as I'm concerned, if the answer is yes to both, this has much potential. I'm just trying to figure out what I'm looking at) Thank you! [1]: https://blog.openmined.org/duet-demo-how-to-do-data-science-...

Yes, there are many parallels with Duets we can look at Sarus as a productized version of it.

There are some differences though: - we designed for the trusted curator model where Duet is mostly for federated learning tasks in mind - the privacy policies are based on principles (such as: "DP-outputs with epsilon < 2 can be shared", "DP-synthetic data can be shared", or "weights of ML models can be shared"), then the gateway applies the principles to any query, whether it is a SQL query, an ML model or else. In Duet, it's all about manual validation of given queries.

Post reply on HN