Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL Anonymizer

postgresql-anonymizer.readthedocs.io

Post reply on HN
Print view View original

31–40 of 54 posts

Re: PostgreSQL Anonymizer

#31

Post by Cynddl »

I'm going to repeat myself as I do everytime I encounter such tools. These tools DO NOT provide anonymization, and especially not at the level required by the EU's GDPR (where the notion of PII does not exist).

As a computer scientist and academic researcher having worked on this topic for now more than a decade (some of my work if you are interested: [1, 2]), re-identification is often possible from few pieces of information. Masking or replacing a few values or columns will often not provide sufficient guarantees—especially when a lot of information is being released.

What this tool does is called ‘pseudonymization’ and maybe, if not very carefully, ‘de-identification’ in some case. With colleagues, reviewed all the literature and industry practices a few months ago [3], and our conclusion was:

> We find that, although no perfect solution exists, applying modern techniques while auditing their guarantees against attacks is the best approach to safely use and share data today.

This is clearly not what this tool is doing.

[1] https://www.nature.com/articles/s41467-019-10933-3 [2] https://www.nature.com/articles/s41467-024-55296-6 [3] https://www.science.org/doi/10.1126/sciadv.adn7053

Re: PostgreSQL Anonymizer

#32

Post by edrenova »

symfoniq wrote: post #17
dandiep wrote: post #5

This is a fantastic idea. Now how to get it on RDS…

Same. Lack of RDS support is the only reason we aren’t using this.

Just to jump in here -> We support RDS + more and you can self-host, Neosync.

https://github.com/nucleuscloud/neosync

(I'm one of the co-founders)

Re: PostgreSQL Anonymizer

#33

Post by aeontech »

bux93 wrote: post #13
gkbrk wrote: post #12

Clickhouse has something similar called clickhouse-obfuscator [1]. It even works offline with data dumps so you can quickly prepare and send somewhat realistic example data to others. According to its --help output, it is designed to retain the following properties of data: - cardinalities of values (number of distinct values) for every column and for every tuple of columns; - conditional cardinalities: number of dis…

The Dutch national office of statistics has tools intended to de-identify 'microdata' such that k-anonimity[1] is achieved called mu-argus[2] and tau-argus. [1] A release of data is said to have the k-anonymity property if the information for each person contained in the release cannot be distinguished from at least k-1 individuals whose information also appear in the release. https://en.wikipedia.org/wiki/K-anonymit…

This is really cool, and deserves a submission of its own, I'd say!

Re: PostgreSQL Anonymizer

#34

Post by BoorishBears »

edrenova wrote: post #32
symfoniq wrote: post #17

Earlier quoted context omitted.

Same. Lack of RDS support is the only reason we aren’t using this.

Just to jump in here -> We support RDS + more and you can self-host, Neosync. https://github.com/nucleuscloud/neosync (I'm one of the co-founders)

I tried to figure out how/if this does what I need and your README had no examples. I clicked a couple of level deep, found no obvious demonstrations and left.

I checked the homepage but I do not watch Loom-style demos personally, definitely not 5 minute ones, and so I left.

-

When I click on OP's link, or just search for it on Google, it takes less than a full page for the extension to show me an extremely straightforward demonstration of its value. You should have something like that.

A simple example of what queries will look like, what setup will look like, all concisely communicated, no 5 minute lectures involved.

Re: PostgreSQL Anonymizer

#35

Post by aquilaFiera »

Cynddl wrote: post #31

I'm going to repeat myself as I do everytime I encounter such tools. These tools DO NOT provide anonymization, and especially not at the level required by the EU's GDPR (where the notion of PII does not exist). As a computer scientist and academic researcher having worked on this topic for now more than a decade (some of my work if you are interested: [1, 2]), re-identification is often possible from few pieces of in…

Seems like if you're doing static masking and you mask enough data, this works just great. Am I missing something?

Re: PostgreSQL Anonymizer

#36

Post by sgt »

heeton wrote: post #16
lovasoa wrote: post #4

Earlier quoted context omitted.

The principle of the software seems to be that the original data is never altered. It is a postgres extension that "masks" the data for certain postgres users. You can always connect as the root user and see everything when you need to.

It allows updating the original data - https://postgresql-anonymizer.readthedocs.io/en/stable/stati... > These methods will destroy the original data. Use with care.

So basically running SELECT anon.anonymize_database(); will do it.

Re: PostgreSQL Anonymizer

#37

Post by joram87 »

I've been working on something similar, starting a company around the idea! We realized that a lot of people had concerns or challenges with installing an extension on their production database and also that they wanted non-technical folks in compliance or HR to be able to configure and maintain the rules for individual employees. pgAnonymizer is a database extension but we structured ours to be a proxy server that hides/anonymizes/filters the data. We made a web dashboard that simplifies the configuration process, and allows you to configure what to do if columns get added to the database (default mask or hide new columns). We're about to go GA and if anyone has any feedback or wants to a free beta testing trial, I'd love to chat

Re: PostgreSQL Anonymizer

#38

Post by joram87 »

joram87 wrote: post #37

I've been working on something similar, starting a company around the idea! We realized that a lot of people had concerns or challenges with installing an extension on their production database and also that they wanted non-technical folks in compliance or HR to be able to configure and maintain the rules for individual employees. pgAnonymizer is a database extension but we structured ours to be a proxy server that h…

I probably should mention the name of the tool I'm working on... oops, it's VeilStream :)

Re: PostgreSQL Anonymizer

#39

Post by gmassman »

This is a very handy postgres extension! We've been using it at my job for a couple years now to generate test datasets for developers. We have a weekly job that restores a prod backup to a temporary DB, installs the `anon` extension, and runs pg_dump with the masking rules. Overall we've been very happy with this workflow since it gives us a very good idea of how new features will work with our production data. The masking rules do need maintenance as our DB schema changes, but that's par for the course with these kinds of dev tools.

All that said, I wouldn't rely on this extension as a way to deliver anonymized data to downstream consumers outside of our software team. As others have pointed out, this is really more of a pseudonymization technique. It's great for removing phone numbers, emails, etc. from your data set, but it's not going to eradicate PII. Pretty much all anonymized records can be traced back to their source data through PKs or FKs.

Re: PostgreSQL Anonymizer

#40

Post by michelpp »

foreigner wrote: post #18

TIL that PostgreSQL has SECURITY LABEL! It seems like this could be useful for storing all sorts of metadata about database objects, not just security stuff. E.g. like the COMMENT but not global. From reading the docs it looks like you need a "label provider" to get it to work though. I can only seem to find a few label providers around, does anyone know of one that isn't security/anonymization related and could be u…

Yep, it's a completely generic feature, and I'd argue the keyword 'SECURITY' is a noise word and 'LABEL' by itself would be even better, but that's what we're stuck with. As the documentation states: " PostgreSQL places no restrictions on whether or how a label provider must interpret security labels; it merely provides a mechanism for storing them."
Post reply on HN