Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: A tool to seed your dev database with real data

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 79 posts

Re: Show HN: A tool to seed your dev database with real data

#51

Post by cpeterso »

closeparen wrote: post #34
exdsq wrote: post #32

Earlier quoted context omitted.

Does it need to be statistically faithful or can it just fuzz the data type? Maybe that’s better for testing anyway?

* Data that is tightly clustered on certain keys and widely dispersed on other keys can hit some "fun" interactions with sharding regimes, indexes, etc. that random data doesn't. * Brute forcing a whole bunch of invalid values can be a lot less interesting than lighting up unconventional combinations of valid values. * Sometimes you're wrong about the validation rules, i.e. you think you know the allowable enum value…

Since generating valid values is difficult, it sounds like an opportunity for an open-source fuzzy data generator with rules for different database types. Encode all the lessons shared in those "falsehoods programmers believe" articles about names, addresses, phone numbers, email addresses, credit card numbers, dates, time zones, etc into one testing tool that everyone can use.

I don't work with databases, so maybe this already exists. :)

Re: Show HN: A tool to seed your dev database with real data

#52

Post by onion2k »

evoxmusic wrote: post #49
onion2k wrote: post #46

Earlier quoted context omitted.

I consider it potentially harmful. Anonymizing data is a hard problem, and what is considered sensitive is not settled. For example, an IP address is personal identifiable information under the GDPR. Most people don't mask that in their logs though. If you copy records from production that have network information in them (last known IP for example) then your data controller should be very concerned. Another major pr…

It makes sense to me and that's why: 1. Auto-detection of sensitive data is planned 2. Detecting database schema change is also plan to prevent leaking sensitive data. RepliByte responds to a very common need that almost every company end to build internally. The idea is to collaboratively work on a tool that can be used by anyone and that can be improved to avoid leaking data.

Those are great features to have but they're not in the app yet. This is why there should be a warning in the README to tell users to be careful when they use it now.

Re: Show HN: A tool to seed your dev database with real data

#53

Post by sverhagen »

dvt wrote: post #14
kodah wrote: post #8

Earlier quoted context omitted.

Credentials in a config file can be mistakenly checked into a repository. They're easy to exfiltrate from files, say I write a script with well known configuration locations for thousands of applications and just dumbly pull them all from a compromised system. I now have little bits of access to the wider system where I can now jump from system to system. The best way to store ephemeral secrets is in an environment v…

I don't get this logic, that's what .gitignore is for. I've been using .env files for years and never mistakenly checked one into a repo.

Even if you want your credentials in SOME file, you don't want them in THAT file. Because that configuration file contains enough valuable configuration that SHOULD be in Git, in which case the credentials would be in the way.

Re: Show HN: A tool to seed your dev database with real data

#54

Post by time4tea »

From Thoughtworks Tech Radar https://www.thoughtworks.com/radar

21. Production data in test environments Hold We continue to perceive production data in test environments as an area for concern. Firstly, many examples of this have resulted in reputational damage, for example, where an incorrect alert has been sent from a test system to an entire client population. Secondly, the level of security, specifically around protection of private data, tends to be less for test systems. There is little point in having elaborate controls around access to production data if that data is copied to a test database that can be accessed by every developer and QA. Although you can obfuscate the data, this tends to be applied only to specific fields, for example, credit card numbers. Finally, copying production data to test systems can break privacy laws, for example, where test systems are hosted or accessed from a different country or region. This last scenario is especially problematic with complex cloud deployments. Fake data is a safer approach, and tools exist to help in its creation. We do recognize there are reasons for specific elements of production data to be copied, for example, in the reproduction of bugs or for training of specific ML models. Here our advice is to proceed with caution.

Re: Show HN: A tool to seed your dev database with real data

#55

Post by dotancohen »

bottled_poe wrote: post #36
mountainriver wrote: post #23

Earlier quoted context omitted.

This happens literally all the time in large organizations. People make mistakes

And should be easily caught in code reviews and CI jobs?

Which occur after the code is in git and pushed.

Re: Show HN: A tool to seed your dev database with real data

#56

Post by jve »

gregwebs wrote: post #26

Using a customer's production data outside of production probably violates their expectations of your data security practices. I couldn't see myself using this unless there was a mode where only allowed fields are copied and non-id fields are first transformed in a lossy way.

That was the first thing I checked whether it supports data masking. And it does, via transforms, as evoxmusic already pointed out.

However, there may be times when data masking must be nuanced. Suppose some random email/domain pair is bad and you would rather replace all "example.com" domain instances with "fake.com", and not "random1.com", "random2.com", etc (for ML, 3rd party random analysis). Out of the box I don't see it is provided, HOWEVER I see that you can write custom transformer: https://github.com/Qovery/replibyte/tree/main/examples/wasm and fulfill your needs.

Excellent :)

Re: Show HN: A tool to seed your dev database with real data

#57

Post by jve »

nicoburns wrote: post #10

This does sounds kind of useful. On the other hand I performed a similar task just yesterday using the native pg_dump and pg_restore commands, and it only took a couple of hours to setup (and now I have a repeatable script), so this’ll need to be implemented really well to provide value.

Well, moving sensitive data or PII to development environment for any developer eyeballs to see doesn't sound good and may introduce trouble if data leaked. Development env (where random npm dev scripts get run or whatever) doesn't usually get protected as good as production.

Anyways, if you are single or handful developers where everyone get access to prod, you may not care. Still, data hygiene and risk mitigation shouldn't be overlooked.

Re: Show HN: A tool to seed your dev database with real data

#58

Post by jve »

time4tea wrote: post #54

From Thoughtworks Tech Radar https://www.thoughtworks.com/radar 21. Production data in test environments Hold We continue to perceive production data in test environments as an area for concern. Firstly, many examples of this have resulted in reputational damage, for example, where an incorrect alert has been sent from a test system to an entire client population. Secondly, the level of security, specifically around…

Is this a negative or positive comment?

> Fake data is a safer approach, and tools exist to help in its creation.

Because the tool presented is exactly what this quote says.

Re: Show HN: A tool to seed your dev database with real data

#59

Post by gregwebs »

evoxmusic wrote: post #41
gregwebs wrote: post #26

Using a customer's production data outside of production probably violates their expectations of your data security practices. I couldn't see myself using this unless there was a mode where only allowed fields are copied and non-id fields are first transformed in a lossy way.

That's why RepliByte uses S3 to store transformed data. Then the real production data never leave the production environment. https://github.com/Qovery/replibyte/blob/main/docs/DESIGN.md

The doc refers to S3 as an intermediary- it is then going to be loaded in a different DB where all these issues exist. The intermediary adds one more place where data could be leaked.

Re: Show HN: A tool to seed your dev database with real data

#60

Post by krageon »

Unless you can exhaustively guarantee your customer-data containing production data will definitely be transformed into something completely unrecognisable and irreversible (and let's face it, you can never do so - systems change all the time), using this is irresponsible. The fact that the motivation for it is that it is "tedious" to do the right way doesn't exactly inspire confidence, though it is definitely in the spirit of the times.
Post reply on HN