Replibyte – Seed your database with real data
1–10 of 23 posts
Re: Replibyte – Seed your database with real data
#2I think the description in the man entry is better than the one in the README. Other than that, cool tool!
Re: Replibyte – Seed your database with real data
#3How does it keep personal data safe? I had a look at “how it works” and “faqs” but they don’t answer how you keep stuff safe? It also gets uploaded to S3?
I might have missed it, but I need to know exactly where our PII is stored (so not on a dev laptop), how do you know what to replace and what do you do with any info you do replace?
Edit: To answer my own question, via transformers. But that seems to suggest each dev has to keep it up to date with any schema changes etc
(Also some links are broken on GitHub)
Re: Replibyte – Seed your database with real data
#4The default seems to be to store the sanitized dump on S3.
It’s not always available in a professional context. Or might be considered extraction.
Keeping everything local and detailing exactly what goes where and how would be helpful.
Re: Replibyte – Seed your database with real data
#5One feature I’d love to see is a transformer that instead of providing a random value provides a cryptographic one way hash of the data (ie sha2) - that way key uniqueness stays the same (to avoid unique constraints on columns) and also the same value used in one place will match another value in another table after transformation which more accurately reflects the “shape” of the data.
Re: Replibyte – Seed your database with real data
#6The default seems to be to store the sanitized dump on S3. It’s not always available in a professional context. Or might be considered extraction. Keeping everything local and detailing exactly what goes where and how would be helpful.
Also if it's possible to run everything without uploading it to S3. For a smaller time dev with projects in production I would find this really interesting for debugging the production database data, but in development. Uploading it and having it in S3 would needlessly complicate it for me (even though I can understand enterprise customers might prefer it that way)
Re: Replibyte – Seed your database with real data
#7How does it keep personal data safe? I had a look at “how it works” and “faqs” but they don’t answer how you keep stuff safe? It also gets uploaded to S3? I might have missed it, but I need to know exactly where our PII is stored (so not on a dev laptop), how do you know what to replace and what do you do with any info you do replace? Edit: To answer my own question, via transformers. But that seems to suggest each d…
The user tells it what fields need replacing with the yaml config.
Re: Replibyte – Seed your database with real data
#8One feature I’d love to see is a transformer that instead of providing a random value provides a cryptographic one way hash of the data (ie sha2) - that way key uniqueness stays the same (to avoid unique constraints on columns) and also the same value used in one place will match another value in another table after transformation which more accurately reflects the “shape” of the data.
This will not work, at least not if we’re talking PII as it is defined by a Somewhat Sane (TM) privacy legislation.
Sure, passwords and credit card info is obscured with your methodology, but names, dates of birth, sexual orientation, telephone numbers, email and ip will remain unique. This uniqueness is what allows you to potentially identify a person given enough data.
Re: Replibyte – Seed your database with real data
#9One feature I’d love to see is a transformer that instead of providing a random value provides a cryptographic one way hash of the data (ie sha2) - that way key uniqueness stays the same (to avoid unique constraints on columns) and also the same value used in one place will match another value in another table after transformation which more accurately reflects the “shape” of the data.
This will not work, at least not if we’re talking PII as it is defined by a Somewhat Sane (TM) privacy legislation. Sure, passwords and credit card info is obscured with your methodology, but names, dates of birth, sexual orientation, telephone numbers, email and ip will remain unique. This uniqueness is what allows you to potentially identify a person given enough data.
I suppose that what you’d have to do is change the data and then hash it. But once you’ve changed the data it’s no longer PII, so there’s no reason to hash it.
Of course, given enough data that has been changed can potentially allow you to deduce how that data was changed and thus revert it, at which point it would become PII again and you’d have a problem… but that’s probably a fringe scenario
Re: Replibyte – Seed your database with real data
#10One feature I’d love to see is a transformer that instead of providing a random value provides a cryptographic one way hash of the data (ie sha2) - that way key uniqueness stays the same (to avoid unique constraints on columns) and also the same value used in one place will match another value in another table after transformation which more accurately reflects the “shape” of the data.
This will not work, at least not if we’re talking PII as it is defined by a Somewhat Sane (TM) privacy legislation. Sure, passwords and credit card info is obscured with your methodology, but names, dates of birth, sexual orientation, telephone numbers, email and ip will remain unique. This uniqueness is what allows you to potentially identify a person given enough data.
>Sure, passwords and credit card info is obscured with your methodology
Even that's problematic, because there may be code that depends on the data being somewhat "real". Credit cards, for example, may need to pass LUHN tests, or have valid BIN sections, etc.