41–50 of 79 posts
Using a customer's production data outside of production probably violates their expectations of your data security practices. I couldn't see myself using this unless there was a mode where only allowed fields are copied and non-id fields are first transformed in a lossy way.
This project needs a giant heading box in the README stating 3 things; - staging databases that hold data generated from production databases should be considered production data, with the same level of consideration for security and access as production. - staging databases that hold production data are a GDPR violation waiting to happen. Make sure your data controller / lawyers knows exactly what you're doing with…
interesting, however couldn't it detect tables and columns automatically instead of having to specify them in the configuration file? If I understand correctly each table is to be specified by hand. Say I have nearly a hundred tables...
Using a customer's production data outside of production probably violates their expectations of your data security practices. I couldn't see myself using this unless there was a mode where only allowed fields are copied and non-id fields are first transformed in a lossy way.
And not using the chaos of the real world to harden your software before it hits production probably violates their expectations of correctness. It's a real and interesting tension! Software that helps people walk this line well is valuable.
I think this tool looks great!
I appreciate the time and effort you put on to releasing a free and open source tool to help solve a real problem.
Keep up the great work!
This project needs a giant heading box in the README stating 3 things; - staging databases that hold data generated from production databases should be considered production data, with the same level of consideration for security and access as production. - staging databases that hold production data are a GDPR violation waiting to happen. Make sure your data controller / lawyers knows exactly what you're doing with…
Do you consider transformed data in staging harmful? (Transformed data = where all the sensitive data have been hidden)
Another major problem with tools like replibyte is that people use them properly, and then a database schema changes, but people don't update their script to anonymize new tables or columns. Then a few months later someone notices sensitive data has made its way in to staging, and into the backups, and the database dumps devs made to debug things because "it's only staging data, who cares!"
Protecting user data is something that you need to be extremely vigilant about. In my experience, the less access I have to production data the happier I am. Copying it and using it in staging, even if you're careful about it, fills me with dread.
The comments I've read on here seem strangely negative, I don't understand why. I think this tool looks great! I appreciate the time and effort you put on to releasing a free and open source tool to help solve a real problem. Keep up the great work!
Earlier quoted context omitted.
This happens literally all the time in large organizations. People make mistakes
And should be easily caught in code reviews and CI jobs?
So basically if everything works perfectly, you won't have that problem. We know processes don't work perfectly, so it's better to avoid that issue in the first place by never leaving the literal tokens in the config.
Also, even if your CI catches the problem, it's too late - your credentials are now in the repo and in the CI system and likely in your log collector attached to the CI. You'll need to roll them. Same with reviews.
Earlier quoted context omitted.
Do you consider transformed data in staging harmful? (Transformed data = where all the sensitive data have been hidden)
I consider it potentially harmful. Anonymizing data is a hard problem, and what is considered sensitive is not settled. For example, an IP address is personal identifiable information under the GDPR. Most people don't mask that in their logs though. If you copy records from production that have network information in them (last known IP for example) then your data controller should be very concerned. Another major pr…
1. Auto-detection of sensitive data is planned
2. Detecting database schema change is also plan to prevent leaking sensitive data.
Earlier quoted context omitted.
This happens literally all the time in large organizations. People make mistakes
And should be easily caught in code reviews and CI jobs?
You’re not wrong, but this is a whole class of issue that can be avoided for trivial levels of effort.