21–30 of 54 posts
This is a fantastic idea. Now how to get it on RDS…
+1 for RDS support. I have wanted to use this for a while in our production systems. reply
https://dba.stackexchange.com/questions/306661/how-to-instal...
reply
This is a fantastic idea. Now how to get it on RDS…
In my experience PG anonymizer has performance issues when it comes to large queries.
A good use case that comes to mind is using prod data in a retool app or something for your internal team but you want to mask out certain bits.
I’ve been building Neosync [1] to handle more advanced use cases where you want to anonymize data for lower level environments. This is more useful for stage or dev data. Then prod stays completely unexposed to anyone.
It also has a transactional anonymization api too.
It certainly complicates things but it provides an additional security layer of separation between the PII and it's related data. You can provide your end users access to a database without having to worry about them getting access to the "dangerous" data. If they do indeed need access to the data pointed to via the token they can request access to that related database.
This method also provides more performance since you don't need to encrypt the entire database (which is often required when storing PII) and also don't need to add extra security context function calls to every database request.
One of the best ways to handle this sort of thing is to put things like PII in a separate database entirely and replace it with a token in the "main" database. When something like PII actually needs to be retrieved you first retrieve the token and then search the other database for said token to get the real data. It certainly complicates things but it provides an additional security layer of separation between the P…
One of the best ways to handle this sort of thing is to put things like PII in a separate database entirely and replace it with a token in the "main" database. When something like PII actually needs to be retrieved you first retrieve the token and then search the other database for said token to get the real data. It certainly complicates things but it provides an additional security layer of separation between the P…
One of the best ways to handle this sort of thing is to put things like PII in a separate database entirely and replace it with a token in the "main" database. When something like PII actually needs to be retrieved you first retrieve the token and then search the other database for said token to get the real data. It certainly complicates things but it provides an additional security layer of separation between the P…
Clickhouse has something similar called clickhouse-obfuscator [1]. It even works offline with data dumps so you can quickly prepare and send somewhat realistic example data to others. According to its --help output, it is designed to retain the following properties of data: - cardinalities of values (number of distinct values) for every column and for every tuple of columns; - conditional cardinalities: number of dis…
The summary is pretty funny:
> "After trying four methods, I got so tired of this problem that it was time just to choose something, make it into a usable tool, and announce the solution"
I was actually tasked with building essentially this same thing back in 2014 when I was a junior dev for a fintech startup. They needed an anonymized version of prod database suitable for support team to pull up when trying to reproduce bugs. Did this gigantic thing that would stream the db dump into a C++ app and anonymize it on the fly. Took a similar approach to their masking they do here. Fun project. Company sho…