Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL Anonymizer

postgresql-anonymizer.readthedocs.io

Post reply on HN
Print view View original

41–50 of 54 posts

Re: PostgreSQL Anonymizer

#41

Post by daamien »

phoronixrly wrote: post #11

I have some experience with the 'Masking Views' functionality. If you are going to rely on it and specifically in a Rails app, know that it is against conventions and thus is generally inconvenient. This may be the same with any other framework that features DB schema migrations. More specifically the integration of this functionality at a fortunately ex-employer was purposefully kept away from the dev team (no motiv…

PostgreSQL Anonymizer developer here : the problem you are describing here only affects version 1.x

Version 2.0 was released a few days ago with a branch new masking system that does not block database migrations.

Re: PostgreSQL Anonymizer

#42

Post by daamien »

hylaride wrote: post #19
dandiep wrote: post #5

This is a fantastic idea. Now how to get it on RDS…

It can be done manually: https://dba.stackexchange.com/questions/306661/how-to-instal...

PostgreSQL Anonymizer developer here : this was possible with version 0.8 which is now completely unmaintained.

This extension is currently not available on RDS but it is available on many others DBaaS providers : Azure SQL, Google Cloud SQL, Crunchy Bridge, ....

Re: PostgreSQL Anonymizer

#43

Post by daamien »

zdc1 wrote: post #7
dandiep wrote: post #5

This is a fantastic idea. Now how to get it on RDS…

Assuming if it's for a support team or internal users with a lower SLA, I wonder if it's possible to have a small self-hosted PostgreSQL server that basically acts as a shim by holding a foreign-data wrapper connection to the actual RDS instance

This called "Masking Data Wrappers"

https://postgresql-anonymizer.readthedocs.io/en/stable/maski...

Re: PostgreSQL Anonymizer

#44

Post by daamien »

debarshri wrote: post #22
dandiep wrote: post #5

This is a fantastic idea. Now how to get it on RDS…

In RDS, if you cannot use this, you can create masked view and use query rewrite to make it work. In my experience PG anonymizer has performance issues when it comes to large queries.

Version 2.0 was released a few days ago. It's a complete rewrite in Rust.

Performance should be better than with v1.x

Re: PostgreSQL Anonymizer

#45

Post by daamien »

Cynddl wrote: post #31

I'm going to repeat myself as I do everytime I encounter such tools. These tools DO NOT provide anonymization, and especially not at the level required by the EU's GDPR (where the notion of PII does not exist). As a computer scientist and academic researcher having worked on this topic for now more than a decade (some of my work if you are interested: [1, 2]), re-identification is often possible from few pieces of in…

Of course there's no perfect solution for anonymizing a dataset...

The extension offers a large panel of masking functions : some are pseudonymizing functions but others are more destructive. For instance there's large collection of fake data generators ( names, address, phones, etc. )

It's up to the database administrator or the application developer to decide which columns need to be masked and how it should be masked.

In some use cases, pseudonymization is enough and others anonymization is required....

Re: PostgreSQL Anonymizer

#46

Post by daamien »

riffraff wrote: post #10

this seems great. I wonder tho, how do you ensure new columns are masked by default? It seems a safer alternative would be to start with all columns being statically masked and only unveil them selectively. I guess you can add some CI steps when modifying the db to ensure a give column is allowed or masked, but still, would be nice if this was defaulted the other way around.

This is called "Privacy By Default"

https://postgresql-anonymizer.readthedocs.io/en/stable/priva...

Re: PostgreSQL Anonymizer

#47

Post by daamien »

sgt wrote: post #36
heeton wrote: post #16

Earlier quoted context omitted.

It allows updating the original data - https://postgresql-anonymizer.readthedocs.io/en/stable/stati... > These methods will destroy the original data. Use with care.

So basically running SELECT anon.anonymize_database(); will do it.

Both approach are possible

* Static Masking will destroy the authentic data once for all

* Dynamic Masking will only alter the data the "masked users". Regular users will continue to view the real data.

Both techniques have their own advantage depending on your context.

Re: PostgreSQL Anonymizer

#48

Post by daamien »

hylaride wrote: post #20
symfoniq wrote: post #17

Earlier quoted context omitted.

Same. Lack of RDS support is the only reason we aren’t using this.

It can be done manually: https://dba.stackexchange.com/questions/306661/how-to-instal ... reply

PostgreSQL Anonymizer developer here : this was possible with version 0.8 which is now completely unmaintained. This extension is currently not available on RDS but it is available on many others DBaaS providers : Azure SQL, Google Cloud SQL, Crunchy Bridge, ....

Re: PostgreSQL Anonymizer

#49

Post by daamien »

gmassman wrote: post #39

This is a very handy postgres extension! We've been using it at my job for a couple years now to generate test datasets for developers. We have a weekly job that restores a prod backup to a temporary DB, installs the `anon` extension, and runs pg_dump with the masking rules. Overall we've been very happy with this workflow since it gives us a very good idea of how new features will work with our production data. The…

Pseudonymizing functions are just one way to mask the data.

There are many other masking functions that will actually anonymize the data.

And the extension does not force you to respect the foreign keys.

It's really up to you to decide how you want to implement your masking policy

Post reply on HN