Show HN: A tool to seed your dev database with real data

#1

Post by ev0xmusic » Tue, Apr 26, 2022, 9:38 AM UTC

A bunch of developers and myself have created RepliByte - an open-source tool to seed a development database from a production database.

Features:

  - Support data backup and restore for PostgreSQL, MySQL and MongoDB
  - Replace sensitive data with fake data
  - Works on large database (> 10GB) (read Design)
  - Database Subsetting: Scale down a production database to a more reasonable size
  - Start a local database with the prod data in a single command
  - On-the-fly data (de)compression (Zlib)
  - On-the-fly data de/encryption (AES-256)
  - Fully stateless (no server, no daemon) and lightweight binary
  - Use custom transformers

My motivation:

As a developer, creating a fake dataset for running tests is tedious. Plus, it does not reflect the real-world data and painful to keep updated. If you prefer to run your app tests with production data. Then RepliByte is for you as well.

Available for MacOSX, Linux and Windows.

> https://github.com/qovery/replibyte

Re: Show HN: A tool to seed your dev database with real data

#2

Post by mdaniel » Wed, Apr 27, 2022, 4:14 AM UTC

Please don't require static AWS credentials: https://github.com/Qovery/replibyte/blob/v0.4.4/replibyte/sr...

or at least either include "AWS_SESSION_TOKEN" in that setup (if it is present) in order to allow "aws sts assume-role" to work, or allow `AWS_PROFILE`, or just use the aws-sdk's normal credential discovery mechanism which at least on their "main" SDKs is a fallback list of them, but I couldn't follow the docs.rs soup in order to know if their rust sdk is up to speed or what

Re: Show HN: A tool to seed your dev database with real data

#3

Post by herpderperator » Wed, Apr 27, 2022, 9:34 PM UTC

mdaniel wrote: post #2

Please don't require static AWS credentials: https://github.com/Qovery/replibyte/blob/v0.4.4/replibyte/sr... or at least either include "AWS_SESSION_TOKEN" in that setup (if it is present) in order to allow "aws sts assume-role" to work, or allow `AWS_PROFILE`, or just use the aws-sdk's normal credential discovery mechanism which at least on their "main" SDKs is a fallback list of them, but I couldn't follow the docs…

Why?

Re: Show HN: A tool to seed your dev database with real data

#4

Post by jensenbox » Wed, Apr 27, 2022, 9:44 PM UTC

herpderperator wrote: post #3

mdaniel wrote: post #2

Please don't require static AWS credentials: https://github.com/Qovery/replibyte/blob/v0.4.4/replibyte/sr... or at least either include "AWS_SESSION_TOKEN" in that setup (if it is present) in order to allow "aws sts assume-role" to work, or allow `AWS_PROFILE`, or just use the aws-sdk's normal credential discovery mechanism which at least on their "main" SDKs is a fallback list of them, but I couldn't follow the docs…

Why?

Static tokens are terrible. They are far too easy to egress and are downright evil.

Re: Show HN: A tool to seed your dev database with real data

#5

Post by alphabettsy » Wed, Apr 27, 2022, 9:48 PM UTC

herpderperator wrote: post #3

mdaniel wrote: post #2

Please don't require static AWS credentials: https://github.com/Qovery/replibyte/blob/v0.4.4/replibyte/sr... or at least either include "AWS_SESSION_TOKEN" in that setup (if it is present) in order to allow "aws sts assume-role" to work, or allow `AWS_PROFILE`, or just use the aws-sdk's normal credential discovery mechanism which at least on their "main" SDKs is a fallback list of them, but I couldn't follow the docs…

Why?

I presume so that one can use instance roles or automatic credential discovery that is common within AWS and using AWS applications.

Re: Show HN: A tool to seed your dev database with real data

#6

Post by macksd » Wed, Apr 27, 2022, 9:50 PM UTC

herpderperator wrote: post #3

mdaniel wrote: post #2

Please don't require static AWS credentials: https://github.com/Qovery/replibyte/blob/v0.4.4/replibyte/sr... or at least either include "AWS_SESSION_TOKEN" in that setup (if it is present) in order to allow "aws sts assume-role" to work, or allow `AWS_PROFILE`, or just use the aws-sdk's normal credential discovery mechanism which at least on their "main" SDKs is a fallback list of them, but I couldn't follow the docs…

Why?

Some organizations use this feature to enforce security policy and ensure regular rotation of credentials. You can't get a static access key ID / secret access key pair, but you can get one with a session token, and so you run everything inside that context. If you can't specify the session token to an Amazon client, you just can't use Amazon STS.

Re: Show HN: A tool to seed your dev database with real data

#7

Post by moltar » Wed, Apr 27, 2022, 10:17 PM UTC

mdaniel wrote: post #2

Please don't require static AWS credentials: https://github.com/Qovery/replibyte/blob/v0.4.4/replibyte/sr... or at least either include "AWS_SESSION_TOKEN" in that setup (if it is present) in order to allow "aws sts assume-role" to work, or allow `AWS_PROFILE`, or just use the aws-sdk's normal credential discovery mechanism which at least on their "main" SDKs is a fallback list of them, but I couldn't follow the docs…

Second this. I would NEVER hardcode credentials into a config file.

I run all AWS commands through an assumed role (STS) via aws-vault.

Re: Show HN: A tool to seed your dev database with real data

#8

Post by kodah » Wed, Apr 27, 2022, 10:43 PM UTC

herpderperator wrote: post #3

mdaniel wrote: post #2

Please don't require static AWS credentials: https://github.com/Qovery/replibyte/blob/v0.4.4/replibyte/sr... or at least either include "AWS_SESSION_TOKEN" in that setup (if it is present) in order to allow "aws sts assume-role" to work, or allow `AWS_PROFILE`, or just use the aws-sdk's normal credential discovery mechanism which at least on their "main" SDKs is a fallback list of them, but I couldn't follow the docs…

Why?

Credentials in a config file can be mistakenly checked into a repository. They're easy to exfiltrate from files, say I write a script with well known configuration locations for thousands of applications and just dumbly pull them all from a compromised system. I now have little bits of access to the wider system where I can now jump from system to system.

The best way to store ephemeral secrets is in an environment variable or /dev/shm. This locks the secret behind the scope of the parent process (shell instance) and the user.

Re: Show HN: A tool to seed your dev database with real data

#9

Post by osrec » Wed, Apr 27, 2022, 10:52 PM UTC

kodah wrote: post #8

herpderperator wrote: post #3

Earlier quoted context omitted.

Why?

Credentials in a config file can be mistakenly checked into a repository. They're easy to exfiltrate from files, say I write a script with well known configuration locations for thousands of applications and just dumbly pull them all from a compromised system. I now have little bits of access to the wider system where I can now jump from system to system. The best way to store ephemeral secrets is in an environment v…

How do you automate setting that environment variable?

Re: Show HN: A tool to seed your dev database with real data

#10

Post by nicoburns » Wed, Apr 27, 2022, 11:01 PM UTC

This does sounds kind of useful. On the other hand I performed a similar task just yesterday using the native pg_dump and pg_restore commands, and it only took a couple of hours to setup (and now I have a repeatable script), so this’ll need to be implemented really well to provide value.