Am I missing the obvious, why would one seed a dev database from production? If anything, data on dev should exist before production?
Show HN: A tool to seed your dev database with real data
61–70 of 79 posts
Re: Show HN: A tool to seed your dev database with real data
#62Unless you can exhaustively guarantee your customer-data containing production data will definitely be transformed into something completely unrecognisable and irreversible (and let's face it, you can never do so - systems change all the time), using this is irresponsible. The fact that the motivation for it is that it is "tedious" to do the right way doesn't exactly inspire confidence, though it is definitely in the…
I feel I'm missing a reference for the configuration file, on their GitHub page. So maybe this already exists, but it would maybe be useful if there were some features for your concern. Like an opt-out mode where you have to specify transformations for all columns unless indicated otherwise. Or at least for text columns.
There are also plenty of organizations (not mine) with review boards for database changes. Those folks could also have a process to make sure that new, sensitive columns get added to the configuration file.
Re: Show HN: A tool to seed your dev database with real data
#63From Thoughtworks Tech Radar https://www.thoughtworks.com/radar 21. Production data in test environments Hold We continue to perceive production data in test environments as an area for concern. Firstly, many examples of this have resulted in reputational damage, for example, where an incorrect alert has been sent from a test system to an entire client population. Secondly, the level of security, specifically around…
Is this a negative or positive comment? > Fake data is a safer approach, and tools exist to help in its creation. Because the tool presented is exactly what this quote says.
No, the tool presented is to copy real production data down to a test instance with support for anonymization, subsetting, etc. That's a very different approach than tools for creating fake data.
Re: Show HN: A tool to seed your dev database with real data
#64Am I missing the obvious, why would one seed a dev database from production? If anything, data on dev should exist before production?
Once you have data in production it's very possible for the fake data you generate in dev not really matching the sort of data you have in production (either in size or because of assumptions made whilst generating it or even bad app updates/schema migrations in the past causing duff data). It can then be useful for future development or debugging that the data is real(-ish).
Re: Show HN: A tool to seed your dev database with real data
#65I like this but after a cursory glance at the source I have a few concerns:
- There's a transformer which appears to retain the first char on string fields. That's not safe if you're dealing with customer data.
- Remove telemetry. That it's claimed to be anonymized and togglable is meaningless where sensitive data is concerned.
Re: Show HN: A tool to seed your dev database with real data
#66This does sounds kind of useful. On the other hand I performed a similar task just yesterday using the native pg_dump and pg_restore commands, and it only took a couple of hours to setup (and now I have a repeatable script), so this’ll need to be implemented really well to provide value.
I am on the same boat but couple hours is terrible still. The best is probably copying the data directory straight which should cut it down to seconds, but i have yet to automate that + there are production credentials/sensitive data problems that needs to be tackled too...
I suspect it's likely to take a couple of hours to set up this tool too!
Re: Show HN: A tool to seed your dev database with real data
#67This does sounds kind of useful. On the other hand I performed a similar task just yesterday using the native pg_dump and pg_restore commands, and it only took a couple of hours to setup (and now I have a repeatable script), so this’ll need to be implemented really well to provide value.
Well, moving sensitive data or PII to development environment for any developer eyeballs to see doesn't sound good and may introduce trouble if data leaked. Development env (where random npm dev scripts get run or whatever) doesn't usually get protected as good as production. Anyways, if you are single or handful developers where everyone get access to prod, you may not care. Still, data hygiene and risk mitigation s…
Yeah, we have a shared QA environment which gets a fair amount of data populated in it, and this is what I was actually syncing in this case. If syncing prod data then I'd definitely want to have very thorough filtering. But then at that point I'm not sure I'd trust this tool!
Re: Show HN: A tool to seed your dev database with real data
#68This is actually a much harder problem than it seems. GDPR is quite strict about what is considered PII (and rightly so). For example: you may think replacing sensitive data with fake data is enough to anonymise customer data. It's not:
> "Personal data is any information that relates to an identified or identifiable living individual. Different pieces of information, which collected together can lead to the identification of a particular person, also constitute personal data."
So it's not enough to, for example, replace all names, addresses etc. when you can still see which products someone has interacted with, when their account was created (which in the production DB would relate back to their actual account!) or any other unexpected pieces of information that links back to their identity.
In practice, this means that any realistic production-derived data is either very likely to be still considered PII (and therefore much more demanding to handle safely and securely) or has to be mangled so much that it is no longer representative of production data.
Re: Show HN: A tool to seed your dev database with real data
#69Earlier quoted context omitted.
And should be easily caught in code reviews and CI jobs?
Which occur after the code is in git and pushed.
So change the secrets and/or find professional developers? Sorry, not sorry, it’s an amateur mistake, not acceptable in a professional environment. Why would an amateur possess the secrets in the first place?
Re: Show HN: A tool to seed your dev database with real data
#70Earlier quoted context omitted.
Which occur after the code is in git and pushed.
So change the secrets and/or find professional developers? Sorry, not sorry, it’s an amateur mistake, not acceptable in a professional environment. Why would an amateur possess the secrets in the first place?
I prefer a security model slightly more robust than "None of the professionals working on this project will ever make a mistake".