Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: A tool to seed your dev database with real data

news.ycombinator.com

Post reply on HN
Print view View original

71–79 of 79 posts

Re: Show HN: A tool to seed your dev database with real data

#71

Post by mdaniel »

evoxmusic wrote: post #40
mdaniel wrote: post #2

Please don't require static AWS credentials: https://github.com/Qovery/replibyte/blob/v0.4.4/replibyte/sr... or at least either include "AWS_SESSION_TOKEN" in that setup (if it is present) in order to allow "aws sts assume-role" to work, or allow `AWS_PROFILE`, or just use the aws-sdk's normal credential discovery mechanism which at least on their "main" SDKs is a fallback list of them, but I couldn't follow the docs…

You can use env vars in the conf yaml file -> https://github.com/Qovery/replibyte/blob/main/examples/sourc...

That is better, but apologies that I overloaded the word "static" -- I meant the non-assume-role flavor, "aws iam create-access-key", static style credentials. Static in that from AWS's perspective, once issued they don't expire, unlike session credentials that have a finite lifespan

I can now see how my wording was confusing, I'll try to be more precise in the future

Re: Show HN: A tool to seed your dev database with real data

#72

Post by mdaniel »

simtel20 wrote: post #31
mdaniel wrote: post #22

Earlier quoted context omitted.

That was the "I can't grok the docs.rs in order to tell you" part; in boto and the sane SDKs, there's a bunch of "Credential Providers" and they're in a list, meaning some are tried before others I have no idea what implementations there are for this: https://docs.rs/aws-sdk-s3/latest/aws_sdk_s3/struct.Credenti... and its official page is even worse: https://docs.aws.amazon.com/sdk-for-rust/latest/dg/credentia... Goi…

Not sure if this is any better than the docs (which are raw, at best) - I'm a middling beginner rust programmer, but I did do this test[1] for myself and it matches most of the other SDKs. [1] https://github.com/pcn/check-assume-role-s3/blob/master/src/...

That's excellent news, thanks so much for taking the time to put that together!

Re: Show HN: A tool to seed your dev database with real data

#73

Post by xwowsersx »

This looks very useful and I have an immediate need for this. I'm still not sure how to use replibyte to do the following: I want to take a snapshot of the DB from one of my environments and then seed a local DB with it. I see this is a basic use case of replibyte, but not sure exactly how to accomplish this.

I have a docker container running postgres and I just want to take the snapshot and seed it into that. How exactly do I do this?

Re: Show HN: A tool to seed your dev database with real data

#74

Post by evoxmusic »

xwowsersx wrote: post #73

This looks very useful and I have an immediate need for this. I'm still not sure how to use replibyte to do the following: I want to take a snapshot of the DB from one of my environments and then seed a local DB with it. I see this is a basic use case of replibyte, but not sure exactly how to accomplish this. I have a docker container running postgres and I just want to take the snapshot and seed it into that. How ex…

Can you open an issue on the GitHub repo and set label "question" ? Thx

Re: Show HN: A tool to seed your dev database with real data

#75

Post by mountainriver »

dotancohen wrote: post #70
bottled_poe wrote: post #69

Earlier quoted context omitted.

So change the secrets and/or find professional developers? Sorry, not sorry, it’s an amateur mistake, not acceptable in a professional environment. Why would an amateur possess the secrets in the first place?

I prefer a security model slightly more robust than "None of the professionals working on this project will ever make a mistake".

You too?

Re: Show HN: A tool to seed your dev database with real data

#76

Post by ev0xmusic »

tjpnz wrote: post #65

I like this but after a cursory glance at the source I have a few concerns: - There's a transformer which appears to retain the first char on string fields. That's not safe if you're dealing with customer data. - Remove telemetry. That it's claimed to be anonymized and togglable is meaningless where sensitive data is concerned.

Thanks:

  1. What do you mean it is not safe?

  2. Telemetry can be removed with the option --no-telemetry and you can inspect the code > https://github.com/Qovery/replibyte/blob/main/replibyte/src/telemetry.rs

Re: Show HN: A tool to seed your dev database with real data

#77

Post by bprasanna »

One of my colleagues has developed a sophisticated Data generator addressing the needs of workloads/algorithms which work based on the characteristics of the data.

https://github.com/jssprasanna/redgene

ReDGene - Relational Data Generator is a tool aimed at taking control over the data generation with being able to generate column vectors in a table with required type, interval, length, cardinality, skewness and constraints like Primary Key, Foreign Key, Foreign Key(unique 1:1 mapping), Composite Primary Key and Composite Foreign Key.

And this is DB agnostic, it generates data in the form of flat files which can be imported into any database which supports importing data from flat files.

Re: Show HN: A tool to seed your dev database with real data

#78

Post by tjpnz »

ev0xmusic wrote: post #76
tjpnz wrote: post #65

I like this but after a cursory glance at the source I have a few concerns: - There's a transformer which appears to retain the first char on string fields. That's not safe if you're dealing with customer data. - Remove telemetry. That it's claimed to be anonymized and togglable is meaningless where sensitive data is concerned.

Thanks: 1. What do you mean it is not safe? 2. Telemetry can be removed with the option --no-telemetry and you can inspect the code > https://github.com/Qovery/replibyte/blob/main/replibyte/src/telemetry.rs

It's not safe because I could potentially use that information to find a real customer in the DB. It becomes more problematic when working with data from Asian countries where it's possible (even common) for family and/or first names to consist of two or even a single character.

With regards to telemetry I'm aware that it can be disabled. But in my experience that would still result in a veto from the security teams I've worked with.

Re: Show HN: A tool to seed your dev database with real data

#79

Post by carterschonwald »

closeparen wrote: post #34
exdsq wrote: post #32

Earlier quoted context omitted.

Does it need to be statistically faithful or can it just fuzz the data type? Maybe that’s better for testing anyway?

* Data that is tightly clustered on certain keys and widely dispersed on other keys can hit some "fun" interactions with sharding regimes, indexes, etc. that random data doesn't. * Brute forcing a whole bunch of invalid values can be a lot less interesting than lighting up unconventional combinations of valid values. * Sometimes you're wrong about the validation rules, i.e. you think you know the allowable enum value…

Exactly!

A fun example is city and state and country fields in a row!

We tend to see banks or businesses with locations in nyc, ny, USA. And quite unlikely to see a business hq in New York, Hawaii USA. If it even exists.

Post reply on HN