Replibyte – Seed your database with real data • Hacker News Board

Re: Replibyte – Seed your database with real data

#21

Post by zX41ZdbW » Mon, Jul 11, 2022, 8:44 AM UTC

I recommend checking out clickhouse-obfuscator. It's a more sophisticated tool for dataset obfuscation.

Installation (single binary Linux/Mac/FreeBSD):

curl https://clickhouse.com/ | sh

./clickhouse obfuscator --help

Docs: https://clickhouse.com/docs/en/operations/utilities/clickhou...

Re: Replibyte – Seed your database with real data

#22

Post by evoxmusic » Mon, Jul 11, 2022, 9:01 AM UTC

zX41ZdbW wrote: post #21

I recommend checking out clickhouse-obfuscator. It's a more sophisticated tool for dataset obfuscation. Installation (single binary Linux/Mac/FreeBSD): curl https://clickhouse.com/ | sh ./clickhouse obfuscator --help Docs: https://clickhouse.com/docs/en/operations/utilities/clickhou...

I will take a look for Replibyte. Thanks for sharing

Re: Replibyte – Seed your database with real data

#23

Post by BobbyJo » Mon, Jul 11, 2022, 3:15 PM UTC

roskilli wrote: post #5

One feature I’d love to see is a transformer that instead of providing a random value provides a cryptographic one way hash of the data (ie sha2) - that way key uniqueness stays the same (to avoid unique constraints on columns) and also the same value used in one place will match another value in another table after transformation which more accurately reflects the “shape” of the data.

I hate to be so self promoting (I swear I'm just trying to be helpful), but Gretel has that as a transformer you can use[0]. You can test out a lot of our stuff without payment info through our console[1] if you just want to mess around and see if tools like it ( and Replibyte of course :) ) would fit your use case. That being said, you can run into issues using direct transforms like this, depending on the correlated data, because of various known deanonymization attacks. There are some pretty gnarly examples out there if you Google around.

[0]https://docs.gretel.ai/gretel.ai/transforms/transforms-model...

[1]https://console.gretel.cloud/login