Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: A tool to seed your dev database with real data

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 79 posts

Re: Show HN: A tool to seed your dev database with real data

#31

Post by simtel20 »

mdaniel wrote: post #22
atonse wrote: post #15

Earlier quoted context omitted.

Should work out of the box if they’re using the rust AWS library right?

That was the "I can't grok the docs.rs in order to tell you" part; in boto and the sane SDKs, there's a bunch of "Credential Providers" and they're in a list, meaning some are tried before others I have no idea what implementations there are for this: https://docs.rs/aws-sdk-s3/latest/aws_sdk_s3/struct.Credenti... and its official page is even worse: https://docs.aws.amazon.com/sdk-for-rust/latest/dg/credentia... Goi…

Not sure if this is any better than the docs (which are raw, at best) - I'm a middling beginner rust programmer, but I did do this test[1] for myself and it matches most of the other SDKs.

[1] https://github.com/pcn/check-assume-role-s3/blob/master/src/...

Re: Show HN: A tool to seed your dev database with real data

#32

Post by exdsq »

carterschonwald wrote: post #29
exdsq wrote: post #28

You’d imagine Postgres or whatever would have a built in function to populate a DB based on types as a sort of fuzzing tool tbh I worked on a gov app years ago that required anonymized databases and I remember thinking that then - why isn’t it available out the box? Everyone must need this from time to time

I build a dataset anonymizer at jpmorgan years ago. There’s a surprising amount of nuance needed just to do a decent job generating schematically valid fake data, let alone stuff that’s statistically faithful to true data

Does it need to be statistically faithful or can it just fuzz the data type? Maybe that’s better for testing anyway?

Re: Show HN: A tool to seed your dev database with real data

#33

Post by closeparen »

gregwebs wrote: post #26

Using a customer's production data outside of production probably violates their expectations of your data security practices. I couldn't see myself using this unless there was a mode where only allowed fields are copied and non-id fields are first transformed in a lossy way.

And not using the chaos of the real world to harden your software before it hits production probably violates their expectations of correctness. It's a real and interesting tension! Software that helps people walk this line well is valuable.

Re: Show HN: A tool to seed your dev database with real data

#34

Post by closeparen »

exdsq wrote: post #32
carterschonwald wrote: post #29

Earlier quoted context omitted.

I build a dataset anonymizer at jpmorgan years ago. There’s a surprising amount of nuance needed just to do a decent job generating schematically valid fake data, let alone stuff that’s statistically faithful to true data

Does it need to be statistically faithful or can it just fuzz the data type? Maybe that’s better for testing anyway?

* Data that is tightly clustered on certain keys and widely dispersed on other keys can hit some "fun" interactions with sharding regimes, indexes, etc. that random data doesn't.

* Brute forcing a whole bunch of invalid values can be a lot less interesting than lighting up unconventional combinations of valid values.

* Sometimes you're wrong about the validation rules, i.e. you think you know the allowable enum values here but in fact production systems that really exist and have customers behind them are setting other values. Rejecting those would itself be a bug.

Re: Show HN: A tool to seed your dev database with real data

#35

Post by jlgaddis »

I'm (not) looking forward to the future data breach notifications / post-mortems that include something like "... our developers used a tool to copy the production database to a dev database on their laptop ..."

Honestly, I'm kinda surprised by the lack of comments advocating against doing this.

Re: Show HN: A tool to seed your dev database with real data

#36

Post by bottled_poe »

mountainriver wrote: post #23
dvt wrote: post #14

Earlier quoted context omitted.

I don't get this logic, that's what .gitignore is for. I've been using .env files for years and never mistakenly checked one into a repo.

This happens literally all the time in large organizations. People make mistakes

And should be easily caught in code reviews and CI jobs?

Re: Show HN: A tool to seed your dev database with real data

#37

Post by onion2k »

This project needs a giant heading box in the README stating 3 things;

- staging databases that hold data generated from production databases should be considered production data, with the same level of consideration for security and access as production.

- staging databases that hold production data are a GDPR violation waiting to happen. Make sure your data controller / lawyers knows exactly what you're doing with production data.

- ask yourself why you need production data in staging in the first place. What are you gaining over a script that generates data? If you want data at scale you can generate it randomly. If you want data that covers all edge cases you can generate it non-randomly. If you want "real-looking" data then maybe this tool is useful.

People copying data from production to staging and then failing to look after it properly is a nightmare. It shouldn't be encouraged except in very unusual circumstances. In my experience of dev, your development and staging data should be covering the weird edge cases that you need to handle far more than the nice "happy path" data you get in production.

Re: Show HN: A tool to seed your dev database with real data

#38

Post by dvasdekis »

I was thinking "oh! this is awesome!", and then noticed it didn't support MSSQL. Not to worry, I'll just contribute a connector. Let's take a look at their existing connector code...

https://github.com/Qovery/replibyte/blob/main/replibyte/src/...

Not a single comment to say what anything does. Sigh. It's the same for the other drivers too.

Re: Show HN: A tool to seed your dev database with real data

#39

Post by evoxmusic »

dvasdekis wrote: post #38

I was thinking "oh! this is awesome!", and then noticed it didn't support MSSQL. Not to worry, I'll just contribute a connector. Let's take a look at their existing connector code... https://github.com/Qovery/replibyte/blob/main/replibyte/src/... Not a single comment to say what anything does. Sigh. It's the same for the other drivers too.

Can you open an issue to ask for getting better documentation on how to contribute for an additional connector?

Re: Show HN: A tool to seed your dev database with real data

#40

Post by evoxmusic »

mdaniel wrote: post #2

Please don't require static AWS credentials: https://github.com/Qovery/replibyte/blob/v0.4.4/replibyte/sr... or at least either include "AWS_SESSION_TOKEN" in that setup (if it is present) in order to allow "aws sts assume-role" to work, or allow `AWS_PROFILE`, or just use the aws-sdk's normal credential discovery mechanism which at least on their "main" SDKs is a fallback list of them, but I couldn't follow the docs…

You can use env vars in the conf yaml file -> https://github.com/Qovery/replibyte/blob/main/examples/sourc...
Post reply on HN