Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: A tool to seed your dev database with real data

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 79 posts

Re: Show HN: A tool to seed your dev database with real data

#21

Post by fwip »

dvt wrote: post #14
kodah wrote: post #8

Earlier quoted context omitted.

Credentials in a config file can be mistakenly checked into a repository. They're easy to exfiltrate from files, say I write a script with well known configuration locations for thousands of applications and just dumbly pull them all from a compromised system. I now have little bits of access to the wider system where I can now jump from system to system. The best way to store ephemeral secrets is in an environment v…

I don't get this logic, that's what .gitignore is for. I've been using .env files for years and never mistakenly checked one into a repo.

Defense in depth. It only takes one person on your team accidentally committing one file for one service before your .gitignore safeguard is no longer guarding anything.

Re: Show HN: A tool to seed your dev database with real data

#22

Post by mdaniel »

atonse wrote: post #15
mdaniel wrote: post #2

Please don't require static AWS credentials: https://github.com/Qovery/replibyte/blob/v0.4.4/replibyte/sr... or at least either include "AWS_SESSION_TOKEN" in that setup (if it is present) in order to allow "aws sts assume-role" to work, or allow `AWS_PROFILE`, or just use the aws-sdk's normal credential discovery mechanism which at least on their "main" SDKs is a fallback list of them, but I couldn't follow the docs…

Should work out of the box if they’re using the rust AWS library right?

That was the "I can't grok the docs.rs in order to tell you" part; in boto and the sane SDKs, there's a bunch of "Credential Providers" and they're in a list, meaning some are tried before others

I have no idea what implementations there are for this: https://docs.rs/aws-sdk-s3/latest/aws_sdk_s3/struct.Credenti... and its official page is even worse: https://docs.aws.amazon.com/sdk-for-rust/latest/dg/credentia...

Going all the way down to the GH readme seems to back up the investigation that, no, they really seem to have forgotten about "AWS_SESSION_TOKEN": https://github.com/awslabs/aws-sdk-rust#getting-started-with...

Re: Show HN: A tool to seed your dev database with real data

#23

Post by mountainriver »

dvt wrote: post #14
kodah wrote: post #8

Earlier quoted context omitted.

Credentials in a config file can be mistakenly checked into a repository. They're easy to exfiltrate from files, say I write a script with well known configuration locations for thousands of applications and just dumbly pull them all from a compromised system. I now have little bits of access to the wider system where I can now jump from system to system. The best way to store ephemeral secrets is in an environment v…

I don't get this logic, that's what .gitignore is for. I've been using .env files for years and never mistakenly checked one into a repo.

This happens literally all the time in large organizations. People make mistakes

Re: Show HN: A tool to seed your dev database with real data

#24

Post by belthesar »

dvt wrote: post #14
kodah wrote: post #8

Earlier quoted context omitted.

Credentials in a config file can be mistakenly checked into a repository. They're easy to exfiltrate from files, say I write a script with well known configuration locations for thousands of applications and just dumbly pull them all from a compromised system. I now have little bits of access to the wider system where I can now jump from system to system. The best way to store ephemeral secrets is in an environment v…

I don't get this logic, that's what .gitignore is for. I've been using .env files for years and never mistakenly checked one into a repo.

Tell that to Solarwinds.

Re: Show HN: A tool to seed your dev database with real data

#25

Post by Aeolun »

herpderperator wrote: post #3
mdaniel wrote: post #2

Please don't require static AWS credentials: https://github.com/Qovery/replibyte/blob/v0.4.4/replibyte/sr... or at least either include "AWS_SESSION_TOKEN" in that setup (if it is present) in order to allow "aws sts assume-role" to work, or allow `AWS_PROFILE`, or just use the aws-sdk's normal credential discovery mechanism which at least on their "main" SDKs is a fallback list of them, but I couldn't follow the docs…

Why?

One big reason is it’s insecure.

Another big reason is it’s much nicer to deploy on any AWS service and have the SDK use the metadata host, which will automatically provide you with a temporary access token with the permissions of the role you set for it.

Re: Show HN: A tool to seed your dev database with real data

#26

Post by gregwebs »

Using a customer's production data outside of production probably violates their expectations of your data security practices. I couldn't see myself using this unless there was a mode where only allowed fields are copied and non-id fields are first transformed in a lossy way.

Re: Show HN: A tool to seed your dev database with real data

#27

Post by husainfazel » 

  - Works on large database (> 10GB) (read Design)
Can anyone explain to me how this works in RepliByte? The design document only talked about Postgres.

For example let's say I have a MySQL database, how does RepliByte copy that database into S3?

Does it use mysqldump or are they coping the database index files? We have a script that automatically backs up our production database at intervals to S3 and then a program to download the latest backup and scrub the data.

It takes a heck of a long time to download and impacts the server when it happens... it's been on my todo list to replace with Percona's Xtrabackup [1] but doesn't look that's what these guys are doing?

  - Database Subsetting: Scale down a production database to a more reasonable size
What about this? Does the database need foreign keys to prevent related rows in tables being lost and are they just randomly deleting rows as the config seems to indicate [2]

[1] https://www.percona.com/software/mysql-database/percona-xtra...

[2] https://github.com/qovery/replibyte#configuration

Re: Show HN: A tool to seed your dev database with real data

#28

Post by exdsq »

You’d imagine Postgres or whatever would have a built in function to populate a DB based on types as a sort of fuzzing tool tbh

I worked on a gov app years ago that required anonymized databases and I remember thinking that then - why isn’t it available out the box? Everyone must need this from time to time

Re: Show HN: A tool to seed your dev database with real data

#29

Post by carterschonwald »

exdsq wrote: post #28

You’d imagine Postgres or whatever would have a built in function to populate a DB based on types as a sort of fuzzing tool tbh I worked on a gov app years ago that required anonymized databases and I remember thinking that then - why isn’t it available out the box? Everyone must need this from time to time

I build a dataset anonymizer at jpmorgan years ago. There’s a surprising amount of nuance needed just to do a decent job generating schematically valid fake data, let alone stuff that’s statistically faithful to true data

Re: Show HN: A tool to seed your dev database with real data

#30

Post by kristjansson »

dvt wrote: post #14
kodah wrote: post #8

Earlier quoted context omitted.

Credentials in a config file can be mistakenly checked into a repository. They're easy to exfiltrate from files, say I write a script with well known configuration locations for thousands of applications and just dumbly pull them all from a compromised system. I now have little bits of access to the wider system where I can now jump from system to system. The best way to store ephemeral secrets is in an environment v…

I don't get this logic, that's what .gitignore is for. I've been using .env files for years and never mistakenly checked one into a repo.

It’s not just to protect secrets, it’s better architecture. Rely on an ACCESS_ID/SECRET_KEY pair and it’s easy to bind too tightly to that authentication mechanism. Then providing credentials in production is a pain. Use the standard credential provider chain and the transition to production settings is trivial.
Post reply on HN